Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/11936#issuecomment-201300143
@marmbrus Unfortunately our current strategy for generating Spark
partitions doesn't play well with ORC version of `buildReader()`, or to be more
specific, `OrcRecordReader`. The following Spark shell snippet shows the
problem:
```scala
import org.apache.spark.sql.types._
// Just creates a random file that is large enough
val df = sqlContext.range(1000000).select(
'id cast StringType as 'a,
'id cast StringType as 'b,
'id cast StringType as 'c
)
val path = "/tmp/large.orc"
df.write.mode("overwrite").orc(path)
sqlContext.sql(s"SET spark.sql.files.maxPartitionBytes=${1024 * 1024}")
sqlContext.sql(s"SET spark.sql.parquet.enableVectorizedReader=false")
sqlContext.read.orc(path).count() // <-- Gives 500,000 instead of 1,000,000
```
Please refer to discussion [here][1] for details.
[1]: https://github.com/apache/spark/pull/11646/files#r57441074
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
