[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 Yep. @kiszk . @mgaido91 also reports that, so I'm investigating that more. However, that doesn't mean this approach is not proper. You can see the manual test case example in previous ORC-related PR and this PR. This approach definitely reduces the number of point of failures. For the remaining issue, I think we may need a different approach in a different code path. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 For the following, I'll. > Would it be worth to add this JIRA number in a comment as we did for ORC? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20619 Umm, we still see the following exception in [the log](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87523/testReport/) ... ``` Caused by: sbt.ForkMain$ForkError: java.lang.Throwable: null at org.apache.spark.DebugFilesystem$.addOpenStream(DebugFilesystem.scala:36) at org.apache.spark.DebugFilesystem.open(DebugFilesystem.scala:70) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766) at org.apache.orc.impl.RecordReaderUtils$DefaultDataReader.open(RecordReaderUtils.java:173) at org.apache.orc.impl.RecordReaderImpl.(RecordReaderImpl.java:254) at org.apache.orc.impl.ReaderImpl.rows(ReaderImpl.java:633) at org.apache.spark.sql.execution.datasources.orc.OrcColumnarBatchReader.initialize(OrcColumnarBatchReader.java:140) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anonfun$buildReaderWithPartitionValues$2.apply(OrcFileFormat.scala:197) at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anonfun$buildReaderWithPartitionValues$2.apply(OrcFileFormat.scala:161) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:125) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:179) at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:106) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.scan_nextBatch$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408) at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1834) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1162) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1162) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2063) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2063) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) ... 3 more ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87523/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87523/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/946/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87523/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20619 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20619 Would it be worth to add this JIRA number in a comment as we did for ORC? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87520/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87520/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 Thank you for retriggering, @gatorsmile . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87520/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/944/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20619 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87518/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87518/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/943/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87518/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 The failure is irrelevant to this PR. ``` org.apache.spark.sql.hive.client.HiveClientSuites.(It is not a test it is a sbt.testing.NestedSuiteSelector) ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 Oh.. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87516/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87516/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20619 He is already on vacation. : ) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 Thank you for last-minute review before vacation. :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20619 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/942/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20619 **[Test build #87516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87516/testReport)** for PR 20619 at commit [`e08d06c`](https://github.com/apache/spark/commit/e08d06c0e6c0cf23178d12baaa5eb00d55f9b456). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20619 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20619: [SPARK-23390][SQL] Register task completion listeners fi...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20619 The reproducible test case is added into PR description and the code is updated according to @kiszk and @cloud-fan 's comments. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
