GitHub user MaxGekk opened a pull request:
https://github.com/apache/spark/pull/21798
[SPARK-24836][SQL] New option for Avro datasource - ignoreExtension
## What changes were proposed in this pull request?
I propose to add new option for AVRO datasource which should control
ignoring of files without `.avro` extension in read. The option has name
`ignoreExtension` with default value `true`. If both options `ignoreExtension`
and `avro.mapred.ignore.inputs.without.extension` are set, `ignoreExtension`
overrides the former one. Here is an example of usage:
```
spark
.read
.option("ignoreExtension", false)
.avro("path to avro files")
```
## How was this patch tested?
I added a test which checks the option directly and a test for checking
that new option overrides hadoop's config.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/MaxGekk/spark-1 avro-ignore-extension
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21798.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21798
----
commit cdb26f27e83c5340657c967b0b1bc72373e8e606
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-17T15:18:33Z
Adding new option - ignoreExtension
commit d07ced332ba04ac18f270962d887f1e19e9155ec
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-17T15:23:32Z
Revert tests for hadoop's config
commit e86d231ecaef45b816917517482732ab010ec9d8
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-17T19:10:06Z
Adding the ignoreExtension option
commit 565e599cebceaec706ecd1c3c80a4e0c31ce3658
Author: Maxim Gekk <maxim.gekk@...>
Date: 2018-07-17T19:17:19Z
Ticket's number to tests titles is added
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
