Abacn commented on issue #25114: URL: https://github.com/apache/beam/issues/25114#issuecomment-1416646555
Hi @alexmreis sorry if there is any misunderstanding, #22809 is closed because the issue on KafkaIO side is fixed, by #24205 (it comments closes #22809: https://github.com/apache/beam/pull/24205#issuecomment-1353257737) That said, the use case of Dataflow Runner should be fixed in upcoming Beam v2.45.0 It still experiencing issues on portable runner (flink, direct streaming) an issue not limited to kafka source, it is that the "splittable DoFn" streaming source not yet supported by portable runner (#20979). I also got bite by this issue quite often (when I validating the fix of #24205, see comments of #22809 I had). The gap between Dataflow and local runners is definitely an important thing need improve. This has direct impact to developers. Besides, no unit test in Python Kafka IO is intended. Within the cross-language framework, the code running kafka read is Java's KafkaIO and unit test is exercised there. We have CrossLanguage Validation Runner (XVR) Tests for each xlang IO and each SDK exercised in schedule. And I recently added a Python KafkaIO performance test also. That said KafkaIO in both Java and Python are our team's priority. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
