ifndef-SleePy commented on pull request #1801: URL: https://github.com/apache/incubator-inlong/pull/1801#issuecomment-970214335
Hi leo, thanks for contributing! Sorry for late response. Currently Sort module is based on Flink DataStream API. And all connectors are reimplemented. The reason here, is the scenario of Sort is much more complicated than Flink SQL implementation. Some features, like schema dynamically updating, data metric collection, multi-tenant supporting, error handling, data quality verification are not easy to support through Flink SQL. Actually in Tencent, we do have a platform running Flink SQL and other APIs that consumes data from TubeMQ or Pulsar. However it's not a part of InLong project. It's a highly customized platform as an internal infrastructure for Sort. However it's not open-source yet, so we changed Sort to depend on Flink Standalone cluster in InLong open-source version. Anyway, it's hard to give this proposal an appropriate position. It's more like a parallel component to Sort. Maybe we could file an issue to discuss whether we should support another consumer parallel to Sort or not? What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
