skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-497625796
@dongjoon-hyun I do respect other people's opinions, it is not the first
time
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-496030795
@dongjoon-hyun Do you also think that the code base is too large so adding
a few
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494772544
@dongjoon-hyun what is your call?
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494550083
> Spark already has a large code base. Since this can be done easily inside
a
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494520497
@zsxwing
> The inconsistency between Spark's checkpoint and Kafka commit
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494326600
> That is also right but then we need to warn it for
commitOffsetsOnCheckpoints
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494324725
I understand the danger it is the same danger as with `kafka.group.id` but I
was
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494320693
> One thing we can imagine is running multiple queries in same application -
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494313373
> Something like monitoringGroupId
Using the source options affects all the
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-494092510
@gaborgsomogyi @HeartSaVioR what if I introduce queryOptions at the
`query.start()`
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-493974211
> Why not use one group and listConsumerGroupOffsets?
@gaborgsomogyi yes
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-493964039
I will update the PR shortly addressing all above comments, already have
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-493961670
@gaborgsomogyi
> A high level design question.
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-492806050
@arunmahadevan I havent tried it but I know commits can be [out of
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-492766741
@arunmahadevan regarding the other issue about the failing batch I think
since the
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-492733735
@arunmahadevan yes we have thought of this, my colleague @seglo has created
a tool
skonto commented on issue #24613: [SPARK-27549][SS] Add support for committing
kafka offsets per batch for supporting external tooling
URL: https://github.com/apache/spark/pull/24613#issuecomment-492616494
@gaborgsomogyi thanks, havent faced any issues. I will check if I can add
unit
17 matches
Mail list logo
