xBis7 commented on PR #68082: URL: https://github.com/apache/airflow/pull/68082#issuecomment-4691808034
Hey @jason810496, I've updated the PR. It took me more time than originally planned but I need to try a few different approaches and test things thoroughly. I think that this last revision is better. I added two new options in the listener config 1. set the kafka connection name 2. set a json dictionary with the kafka config The producer, consumer and admin client hooks, all require an airflow connection to be initialized. The airflow connection carries an extra param JSON that holds all the necessary configuration. This configuration can also carry security info. Initially I tried to workaround it by defining `bootstrap.servers` in the listener config, so that users can initialize a producer without creating a connection. Confluent kafka producer accepts the config as a dictionary. But the airflow connection json is a string, which has limitation like not being able to pass standard callables like `on_delivery` or `oauth_cb`, etc. without having to hardcode them in the hook or the caller. I added an optional parameter to the base class, so that hook callers can pass the config dictionary directly to the hook without using a connection. The param is optional for back-compat reasons. Also, I added a new method that interprets the config callables whether they are part of the connection or the dictionary. It accepts dot-path strings pointing to locally installed packages. If a user has a connection but has also set a config key in the dict param, the dict value overrides the value from the connection. The changes are more extensive than originally planned but I think that they improve the way the hooks are configured greatly. Please take a look and let me know how it looks. I also updated all the tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
