[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20584 I created a PR, https://github.com/apache/spark/pull/20590 . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20584 For the following case, I'll make a PR for Spark ORC columnar reader very soon. > 2) the orc columnar reader's close method doesn't close the file stream. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20584 This patch helps `sbt/hadoop2.7`. So, I'm seriously monitoring the latest consecutive failures at `sbt` and `hadoop-2.6` branch, too. - 4210 (Running) - 4209 Failed with **`FileBasedDataSourceSuite`** and `ParquetQuerySuite` - 4208 **This patch landed here** but failed with `StreamingOuterJoinSuite` and `ReceiverSuite`. - 4207 Failed with `ParquetQuerySuite` - 4206 Failed with `BufferHolderSparkSubmitSuite` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20584 My bad. Thank you, guys. For the following, I'll investigate it. > According to the log, the leaked file stream was created when building the ORC columnar reader. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20584 Great! https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/ becomes green again!!! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20584 I think I rushed to take a look at the first time. Thanks for fixing this. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20584 > I am suspicious about relationship between afterEach() and addTaskCompletionListener (call close()). But, not sure. Let us try this approach first. This is one of my speculations. There 2 possibilities I can think of: 1) the task completion listener is not called before `afterEach`. 2) the orc columnar reader's `close` method doesn't close the file stream. For 1), seems we've fixed it in https://github.com/apache/spark/commit/c5a31d160f47ba51bb9f8a4f3141851034640fc7 . For 2), I'm not sure and may need help from ORC folks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/20584 merging this to master/2.3. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20584 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87321/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20584 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20584 **[Test build #87321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87321/testReport)** for PR 20584 at commit [`51bb48a`](https://github.com/apache/spark/commit/51bb48a4189aeb0322dd4ccd0f02416a52e963c3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/20584 I am also thinking about this. I agree with this. > According to the log, the leaked file stream was created when building the ORC columnar reader. I am suspicious about relationship between `afterEach()` and `addTaskCompletionListener` (call `close()`). But, not sure. Let us try this approach first. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user sameeragarwal commented on the issue: https://github.com/apache/spark/pull/20584 LGTM, seems plausible! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20584 You are right. I have run out of ideas. LGTM too for a try if it happens more frequently in spark-branch-2.3-test-sbt-hadoop-2.7. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20584 > BTW, my rough wild guess was that case 2. (reading it but not closing it) happens in schema inference path. According to the log, the leaked file stream was created when building the ORC columnar reader. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20584 I won't get in the way but I am less sure on this. I thought this is also flaky in PR builder too anyway. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20584 LGTM, I would merge this first and see whether this can help fix the flaky tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20584 BTW, my rough wild guess was that case 2. (reading it but not closing it) happens in schema inference path. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20584 BTW, I would bet for case 2. in the PR description (just a rough wild guess). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20584 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20584 **[Test build #87321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87321/testReport)** for PR 20584 at commit [`51bb48a`](https://github.com/apache/spark/commit/51bb48a4189aeb0322dd4ccd0f02416a52e963c3). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20584 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/802/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20584: [SPARK-23390][SQL] Flaky Test Suite: FileBasedDataSource...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20584 cc @sameeragarwal @dongjoon-hyun @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
