Github user c-horn commented on the issue:
https://github.com/apache/spark/pull/21676
already resolved by https://github.com/apache/spark/pull/21746
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user c-horn commented on the issue:
https://github.com/apache/spark/pull/21676
Hi @tdas sorry for delay.
My email for github account: chorn4...@gmail.com
This looks fine to me, we can close this PR (and jira ticket) when yours is
merged.
---
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
ping ^^^
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
hey @c-horn , I am ready to merge your PR, and to add you as coauthor i
think i need to know your email address i the github account. Can you provide
me that?
---
Github user c-horn commented on the issue:
https://github.com/apache/spark/pull/21676
@tdas I merged your changes into my branch, test passed, thank you ð
---
-
To unsubscribe, e-mail:
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
Here is my solution based on my suggestion -
https://github.com/apache/spark/pull/21746
I stole your unit test from this PR :) Thank you! I will add you as a
co-author in that PR.
---
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
The offset log contains the watermark value that is going to be used in the
batch corresponding to that offset. For example, "checkpoint/offsets/10" will
contain the watermark value to be used for
Github user c-horn commented on the issue:
https://github.com/apache/spark/pull/21676
I was under the assumption that the offset log contained this data?
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/21676
I think the right solution is to record the updateat watermark in the
commit log, so that the updated watermark can be read back from the commit log
next time the stream is started.
---
Github user c-horn commented on the issue:
https://github.com/apache/spark/pull/21676
Changing `OneTimeExecutor` like this resolves this issue:
```
case class OneTimeExecutor() extends TriggerExecutor {
/**
* Execute a single batch using `batchRunner`.
Github user c-horn commented on the issue:
https://github.com/apache/spark/pull/21676
@tdas @marmbrus
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
11 matches
Mail list logo
