No further voice so far. I'm going to submit a PR. Thanks again for the feedback!
On Mon, Oct 17, 2022 at 9:30 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > Thanks Gabor and Dongjoon for supporting this! > > Bump to reach more eyes. If there is no further voice on this in a couple > of days, I'll consider it as a lazy consensus and submit a PR to this. > > On Sat, Oct 15, 2022 at 3:32 AM Dongjoon Hyun <dongjoon.h...@gmail.com> > wrote: > >> +1 >> >> I agree with Jungtaek and Gabor about switching the default value of >> configurations with the migration guide. >> >> Dongjoon >> >> On Thu, Oct 13, 2022 at 12:46 AM Gabor Somogyi <gabor.g.somo...@gmail.com> >> wrote: >> >>> Hi Jungtaek, >>> >>> Good to hear that the new approach is working fine. +1 from my side. >>> >>> BR, >>> G >>> >>> >>> On Thu, Oct 13, 2022 at 4:12 AM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> I would like to propose flipping the default value of Kafka offset >>>> fetching config. The context is following: >>>> >>>> Before Spark 3.1, there was only one approach on fetching offset, using >>>> consumer.poll(0). This has been pointed out as a root cause for hang since >>>> there is no timeout for metadata fetch. >>>> >>>> In Spark 3.1, we addressed this via introducing a new approach on >>>> fetching offset, via SPARK-32032 >>>> <https://issues.apache.org/jira/browse/SPARK-32032>. Since the new >>>> approach leverages AdminClient and consumer group is no longer needed for >>>> fetching offset, required security ACLs are loosen. >>>> >>>> Reference: >>>> https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#offset-fetching >>>> >>>> There was some concern about behavioral change on the security model >>>> hence we couldn't make the new approach by default. >>>> >>>> During the time, we have observed various Kafka connector related >>>> issues which came from old offset fetching (e.g. hang, issues on rebalance >>>> on customer group, etc.) and we fixed many of these issues via simply >>>> flipping the config. >>>> >>>> Based on this, I would consider the default value as "incorrect". The >>>> security-related behavioral change would be introduced inevitably (they can >>>> set topic based ACL rule), but most people will get benefited. IMHO this is >>>> something we can deal with release/migration note. >>>> >>>> Would like to hear the voices on this. >>>> >>>> Thanks, >>>> Jungtaek Lim (HeartSaVioR) >>>> >>>
