Thanks for reporting the bug! I'll list it as a known issue for 2.4.0 I'm adding my own +1, since all the known blockers are resolved.
On Tue, Oct 30, 2018 at 2:56 PM Xiao Li <lix...@databricks.com> wrote: > Yes, this is not a blocker. > "spark.sql.optimizer.nestedSchemaPruning.enabled" is intentionally off by > default. As DB Tsai said, column pruning of nested schema for Parquet > tables is experimental. In this release, we encourage the whole community > to try this new feature but it might have bugs like what the JIRA > SPARK-25879 reports. > > We still can fix the issues in the minor release of Spark 2.4, as long as > the risk is not high. > > Thanks, > > Xiao > > > On Mon, Oct 29, 2018 at 11:49 PM DB Tsai <dbt...@dbtsai.com.invalid> > wrote: > >> +0 >> >> I understand that schema pruning is an experimental feature in Spark >> 2.4, and this can help a lot in read performance as people are trying >> to keep the hierarchical data in nested format. >> >> We just found a serious bug---it could fail parquet reader if a nested >> field and top level field are selected simultaneously. >> https://issues.apache.org/jira/browse/SPARK-25879 >> >> If we decide to not fix it in 2.4, we should at least document it in >> the release note to let users know. >> >> Sincerely, >> >> DB Tsai >> ---------------------------------------------------------- >> Web: https://www.dbtsai.com >> PGP Key ID: 0x5CED8B896A6BDFA0 >> On Mon, Oct 29, 2018 at 8:42 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: >> > >> > +1 >> > >> > 2018년 10월 30일 (화) 오전 11:03, Gengliang Wang <ltn...@gmail.com>님이 작성: >> >> >> >> +1 >> >> >> >> > 在 2018年10月30日，上午10:41，Sean Owen <sro...@gmail.com> 写道： >> >> > >> >> > +1 >> >> > >> >> > Same result as in RC4 from me, and the issues I know of that were >> >> > raised with RC4 are resolved. I tested vs Scala 2.12 and 2.11. >> >> > >> >> > These items are still targeted to 2.4.0; Xiangrui I assume these >> >> > should just be untargeted now, or resolved? >> >> > SPARK-25584 Document libsvm data source in doc site >> >> > SPARK-25346 Document Spark builtin data sources >> >> > SPARK-24464 Unit tests for MLlib's Instrumentation >> >> > On Mon, Oct 29, 2018 at 5:22 AM Wenchen Fan <cloud0...@gmail.com> >> wrote: >> >> >> >> >> >> Please vote on releasing the following candidate as Apache Spark >> version 2.4.0. >> >> >> >> >> >> The vote is open until November 1 PST and passes if a majority +1 >> PMC votes are cast, with >> >> >> a minimum of 3 +1 votes. >> >> >> >> >> >> [ ] +1 Release this package as Apache Spark 2.4.0 >> >> >> [ ] -1 Do not release this package because ... >> >> >> >> >> >> To learn more about Apache Spark, please see >> http://spark.apache.org/ >> >> >> >> >> >> The tag to be voted on is v2.4.0-rc5 (commit >> 0a4c03f7d084f1d2aa48673b99f3b9496893ce8d): >> >> >> https://github.com/apache/spark/tree/v2.4.0-rc5 >> >> >> >> >> >> The release files, including signatures, digests, etc. can be found >> at: >> >> >> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc5-bin/ >> >> >> >> >> >> Signatures used for Spark RCs can be found in this file: >> >> >> https://dist.apache.org/repos/dist/dev/spark/KEYS >> >> >> >> >> >> The staging repository for this release can be found at: >> >> >> >> https://repository.apache.org/content/repositories/orgapachespark-1291 >> >> >> >> >> >> The documentation corresponding to this release can be found at: >> >> >> https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc5-docs/ >> >> >> >> >> >> The list of bug fixes going into 2.4.0 can be found at the >> following URL: >> >> >> https://issues.apache.org/jira/projects/SPARK/versions/12342385 >> >> >> >> >> >> FAQ >> >> >> >> >> >> ========================= >> >> >> How can I help test this release? >> >> >> ========================= >> >> >> >> >> >> If you are a Spark user, you can help us test this release by taking >> >> >> an existing Spark workload and running on this release candidate, >> then >> >> >> reporting any regressions. >> >> >> >> >> >> If you're working in PySpark you can set up a virtual env and >> install >> >> >> the current RC and see if anything important breaks, in the >> Java/Scala >> >> >> you can add the staging repository to your projects resolvers and >> test >> >> >> with the RC (make sure to clean up the artifact cache before/after >> so >> >> >> you don't end up building with a out of date RC going forward). >> >> >> >> >> >> =========================================== >> >> >> What should happen to JIRA tickets still targeting 2.4.0? >> >> >> =========================================== >> >> >> >> >> >> The current list of open tickets targeted at 2.4.0 can be found at: >> >> >> https://issues.apache.org/jira/projects/SPARK and search for >> "Target Version/s" = 2.4.0 >> >> >> >> >> >> Committers should look at those and triage. Extremely important bug >> >> >> fixes, documentation, and API tweaks that impact compatibility >> should >> >> >> be worked on immediately. Everything else please retarget to an >> >> >> appropriate release. >> >> >> >> >> >> ================== >> >> >> But my bug isn't fixed? >> >> >> ================== >> >> >> >> >> >> In order to make timely releases, we will typically not hold the >> >> >> release unless the bug in question is a regression from the previous >> >> >> release. That being said, if there is something which is a >> regression >> >> >> that has not been correctly targeted please ping me or a committer >> to >> >> >> help target the issue. >> >> > >> >> > --------------------------------------------------------------------- >> >> > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > [image: Spark+AI Summit North America 2019] > <http://t.sidekickopen24.com/s1t/c/5/f18dQhb0S7lM8dDMPbW2n0x6l2B9nMJN7t5X-FfhMynN2z8MDjQsyTKW56dzQQ1-_gV6102?t=https%3A%2F%2Fdatabricks.com%2Fsparkaisummit%2Fnorth-america&si=undefined&pi=406b8c9a-b648-4923-9ed1-9a51ffe213fa> >
