GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21769
[SPARK-24805][SQL] Do not ignore avro files without extensions
## What changes were proposed in this pull request?
In the PR, I propose to change default behaviour of AVRO datasource which
currently ignores files without `.avro` extension in read by default. This PR
sets the default value for `avro.mapred.ignore.inputs.without.extension` to
`false` in the case if the parameter is not set by an user.
## How was this patch tested?
Added a test file without extension in AVRO format, and new test for
reading the file with and wihout specified schema.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 avro-without-extension
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21769.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21769
----
commit 35063ef8e734bdeb39316e02b2e8451d0d75d43a
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-14T09:49:30Z
Test for reading files without avro extension
commit 760f98e7aecb5a4c267599b318479d7f2ade165a
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-14T10:38:33Z
Fix tests
commit 8562a8d43868d551efa6a0e9a00d50cdc838a178
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-14T10:53:24Z
Adding ticket number to test title
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
