HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-525708001
cc. @tdas @zsxwing @jose-torres to see whether committers in this area are
interested on this topic or not.
-
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-525703325
Maybe I need to go through both Beam and Flink to understand the details and
discuss in detailed level.
And @ec
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-525646638
> Beam does not trigger output unless the watermark pass the end of window +
allowed lateness. There is no triggering
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-525225611
> Still there is something I do not understand. With previous Spark Dstream
framework, multiple-aggregations were supp
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-525217328
> Regarding output mode, most of Beam runners (spark for ex) support
discarding output in which element from different
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-524686985
I revisited and thought about this briefly, and felt that the watermark and
mode Spark provide are different with othe
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-501466445
I've left comment in the doc. Sorry I shouldn't leave comment here to make
confusion. Let's talk in doc. Btw I guess t
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-501180447
I just found the case I missed somehow - watermark for (N+1) level of stage
shouldn't refer "input" watermark of N lev
HeartSaVioR commented on issue #23576: [SPARK-26655] [SS] Support multiple
aggregates in append mode
URL: https://github.com/apache/spark/pull/23576#issuecomment-496409067
I'm trying hard to think about the cases which this proposal can bring
correctness issue, and I found nothing. Please
