Github user sbcd90 commented on the pull request:
https://github.com/apache/spark/pull/12774#issuecomment-215927369
Hello @HyukjinKwon , I am able to reproduce the same issue even in Spark
1.6.1. I had two files like this
```
/test_spark/join1.json
{"a": 1, "b": 2}
{"a": 2, "b": 4}
{"a": 4, "b": 8}
{"a": 8, "b": 16}
```
```
/test_spark/subdir/join2.json
{"a": 1, "c": 1}
{"a": 2, "c": 2}
{"a": 3, "c": 3}
{"a": 4, "c": 4}
```
I execute the following code snippet in Spark 1.6.1
```
package org.apache.spark
import org.apache.spark.sql.SQLContext
object TestApp9 extends App {
val conf = new SparkConf().setAppName("TestApp9").setMaster("local")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
sqlContext.read.json("/test_spark").show()
}
```
& the output is
```
+---+---+----+
| a| b| c|
+---+---+----+
| 1| 2|null|
| 2| 4|null|
| 4| 8|null|
| 8| 16|null|
+---+---+----+
```
So, both files are considered. The issue requires further discussion on
what approach to follow to solve it.
The cause of the issue is the piece of code I have changed. But I'm unsure
on what approach to follow to support partitioned tables also.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
