Maxim Gekk created SPARK-24836:
----------------------------------
Summary: New option - ignoreExtension
Key: SPARK-24836
URL: https://issues.apache.org/jira/browse/SPARK-24836
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 2.3.1
Reporter: Maxim Gekk
Need to add new option for Avro datasource - *ignoreExtension*. It should
control ignoring of the .avro extensions. If it is set to *true* (by default),
files with and without .avro extensions should be loaded. Example of usage:
{code:scala}
spark
.read
.option("ignoreExtension", false)
.avro("path to avro files")
{code}
The option duplicates Hadoop's config
avro.mapred.ignore.inputs.without.extension which is taken into account by Avro
datasource now and can be set like:
{code:scala}
spark
.sqlContext
.sparkContext
.hadoopConfiguration
.set("avro.mapred.ignore.inputs.without.extension", "true")
{code}
The ignoreExtension option must override
avro.mapred.ignore.inputs.without.extension.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
