prashanthpdesai edited a comment on issue #661: URL: https://github.com/apache/incubator-hudi/issues/661#issuecomment-632263515
we are trying to use HUDI Deltastreamer to read from compacted Kafka topic in production environment and pull the messages incrementally and persist the data in MapR Platform in regular interval , tried initially with MOR with continuous mode of streamer faced small file issue , now planning to run it in minibatch(every 2 hours) with COPY_ON_WRITE to avoid compaction etc. we are facing offsetoutofrange exception ,tried with both option of auto.offset.reset= earliest as well as latest encountered the same offsetoutofrange exception. we notice that in log that auto.offset.reset is overriding to none WARN kafka010.KafkaUtils: overriding enable.auto.commit to false for executor WARN kafka010.KafkaUtils: overriding auto.offset.reset to none for executor ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
