liuml07 commented on PR #21549: URL: https://github.com/apache/flink/pull/21549#issuecomment-1362629142
Yeah, documenting centrally sounds good. Maybe I'm limited by how I build the jobs with connectors - I shade all connectors (and dependencies) into the uber job jar. For other connectors (e.g. Kafka), I add the dependency to the Flink job following the Maven snippet in each connector's doc page. That works for both local execution (IDE) and remote deployment. Filesystem connector is a bit special because it's in the Flink deploy (so no need to shade) but not ready for local execution. Adding "provided" scope dependency for this connector solves my problem. I don't find other connectors dependency needs to change for local execution. I'm thinking where it would be a good central place. There is a [short guide](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#hadoop-dependencies) for setting Hadoop dependencies for local execution. Do you think it's a good idea to write a new section in the [Connectors and Formats](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/connector/) page or [Advanced Configuration Topics](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/) page? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
