[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15702 I am merging this to master and 2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68637/ Test PASSed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68637 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68637/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68637 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68637/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15702 jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68631/ Test FAILed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68631/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-14 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68631/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68504/ Test PASSed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68504 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68504/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68504 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68504/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15702 jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68496/ Test FAILed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68496/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/15702 LGTM, pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68496/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68029/ Test PASSed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68029/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68029/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68025/ Test FAILed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68025/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68023/ Test FAILed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68023/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68025/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #68023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68023/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #3398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3398/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #3397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3397/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #3398 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3398/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #3397 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3397/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67998/ Test FAILed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67998/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67998/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread koeninger
Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15702 Given the concerns Ofir raised about a single far future event screwing up monotonic event time, do you want to document that problem even if there isn't an enforced filter for it? --- If your

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67939/ Test PASSed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67939 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67939/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67939 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67939/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15702 Not a dumb question! You can certainly use processing time if those are the semantics you require. I do think there is a little bit of work we need to do to ensure determinism for these

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/15702 A very dumb question (I apologize), there is nothing stopping a user to actually use processing time as watermarks with this API either. One can easily do `df.withColumn("timestamp",

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67866/ Test PASSed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed delay based Event Time Watermarks

2016-11-01 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67866/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67866/consoleFull)** for PR 15702 at commit

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread ericl
Github user ericl commented on the issue: https://github.com/apache/spark/pull/15702 I'm still trying to find a failure that includes https://github.com/apache/spark/pull/15701/files. Until then it's hard to debug. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread marmbrus
Github user marmbrus commented on the issue: https://github.com/apache/spark/pull/15702 @ekl - flaky test... Should we turn it off for now? retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67839/ Test FAILed. ---

[GitHub] spark issue #15702: [SPARK-18124] Observed-delay based Event Time Watermarks

2016-10-31 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15702 **[Test build #67839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67839/consoleFull)** for PR 15702 at commit