DeltaStreamer actually just uses the same mechanism as Spark Streaming to manage offsets. So wondering if you see the same behavior with a plain spark streaming job. ?
It manages the offset checkpoints manually by itself within the hoodie commit metadata, to do exactly once ingestion of data.. On Wed, Mar 18, 2020 at 3:07 AM [email protected] <[email protected]> wrote: > > hello, all When using deltastreamer to use kafka data, I want to specify > group.id, but the problem encountered is that after specifying it, I > cannot find it on the kafka side. I found that there are no groups under my > theme. why is it like this? I also manually set enable.auto.commit = true > at the same time, but it didn't seem to work. In kafkaUtils.scala > fixKafkaParams this method, > kafkaParams.put (ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false: > java.lang.Boolean), > which is forced to be rewritten as false. I think it is one of the reasons > why the group cannot be found, but when it is not automatically submitted, > it is usually chosen to be submitted manually. , I did not feel this > phenomenon. > > > Please help answer > thanks > liujinhui > > > [email protected] >
