[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15206 Thanks. Merging to master and 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15206 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65803/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15206 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15206 **[Test build #65803 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65803/consoleFull)** for PR 15206 at commit [`f506a43`](https://github.com/apache/spark/commit/f506a43b5401844e568708cee6a354c4212e3dea). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class FileEntry(path: String, timestamp: Timestamp, batchId: Long) extends Serializable` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15206 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15206 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65801/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15206 **[Test build #65801 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65801/consoleFull)** for PR 15206 at commit [`d9177e5`](https://github.com/apache/spark/commit/d9177e5bd9cd89f70e4f5080587311d58a3a12f8). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class FileEntry(path: String, timestamp: Timestamp, batchId: Long) extends Serializable` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/15206 LGTM, thanks for the fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15206 **[Test build #65803 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65803/consoleFull)** for PR 15206 at commit [`f506a43`](https://github.com/apache/spark/commit/f506a43b5401844e568708cee6a354c4212e3dea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15206 @brkyvz right now we cannot support upgrading anyway since the execution metadata uses Java serialization. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15206 > @zsxwing Will this break streaming jobs if someone upgrades their Spark version, since they won't be able to deserialize the class correctly? @brkyvz No. This doesn't change the file format. `FileEntry(..., batchId = -1)` is only used in `SeenFilesMap` which is only in memory. By the way, people cannot just upgrade the Spark version since `FileStreamSource`'s format has been changed in #13513. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/15206 @zsxwing Will this break streaming jobs if someone upgrades their Spark version, since they won't be able to deserialize the class correctly? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15206 **[Test build #65801 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65801/consoleFull)** for PR 15206 at commit [`d9177e5`](https://github.com/apache/spark/commit/d9177e5bd9cd89f70e4f5080587311d58a3a12f8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15206: [SPARK-17640][SQL]Avoid using -1 as the default batchId ...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/15206 /cc @jerryshao --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
