Re: Flink 1.18.1 cannot read from Kafka

Thanks Biao. Kind regards Phil > On 14 Apr 2024, at 18:04, Biao Geng wrote: > > Hi Phil, > > You can check my github link > > for a detailed tutorial and example codes :). > > Best, > Biao Geng > > Phil

RE: Flink 1.18.1 cannot read from Kafka

Hi Phil, if __name__ == "__main__": process_table() error: link_app | Caused by: org.apache.flink.table.api.ValidationException: Could not find any factory for identifier 'kafka' that implements 'org.apache.flink.table.factories.DynamicTableFactory' in the classpath. flink_app | flink_app |

Re: Flink 1.18.1 cannot read from Kafka

Hi Phil, You can check my github link for a detailed tutorial and example codes :). Best, Biao Geng Phil Stavridis 于2024年4月12日周五 19:10写道： > Hi Biao, > > Thanks for looking into it and providing a detailed example. >

Re: Flink 1.18.1 cannot read from Kafka

Hi Biao, I will check out running with flink run, but should this be run in the Flink JobManager? Would that mean that the container for the Flink JobManager would require both Python installed and a copy of the flink_client.py module? Are there some examples of running flink run in a

Re: Flink 1.18.1 cannot read from Kafka

Hi Phil, It should be totally ok to use `python -m flink_client.job`. It just seems to me that the flink cli is being used more often. And yes, you also need to add the sql connector jar to the flink_client container. After putting the jar in your client container, add codes like

Re: Flink 1.18.1 cannot read from Kafka

Hi Biao, 1. I have a Flink client container like this: # Flink client flink_client: container_name: flink_client image: flink-client:local build: context: . dockerfile: flink_client/Dockerfile networks: - standard depends_on: - jobmanager - Kafka The flink_client/Dockerfile has this bash file

Re: Flink 1.18.1 cannot read from Kafka

Hi Phil, Your codes look good. I mean how do you run the python script. Maybe you are using flink cli? i.e. run commands like ` flink run -t .. -py job.py -j /path/to/flink-sql-kafka-connector.jar`. If that's the case, the `-j /path/to/flink-sql-kafka-connector.jar` is necessary so that in client

Re: Flink 1.18.1 cannot read from Kafka

Hi Biao, For submitting the job, I run t_env.execute_sql. Shouldn’t that be sufficient for submitting the job using the Table API with PyFlink? Isn’t that the recommended way for submitting and running PyFlink jobs on a running Flink cluster? The Flink cluster runs without issues, but there is

Re: Flink 1.18.1 cannot read from Kafka

Hi Phil, Thanks for sharing the detailed information of the job. For you question, how to you submit the job? After applying your yaml file, I think you will successfully launch a flink cluster with 1 JM and 1 TM. Then you would submit the pyflink job to the flink cluster. As the error you showed