Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

2023-10-02 Thread ConradJam
After email communication between me and the author of the warehouse, he has agreed to donate to Apache Flink and participate in the design of the solution. @Martlin @Leonard Yes, it can be said that for clickhouse's connector solution [1] we still refer to part of the code and implementation of

Re: Support AWS SDK V2 for Flink's S3 FileSystem

2023-10-02 Thread David Morávek
Hi Maomao, I wonder whether it would make sense to take a stab at consolidating the S3 filesystems instead and introduce a native one. The whole Hadoop wrapper around the S3 client exists for legacy reasons, and it adds complexity and probably an unnecessary performance penalty. If you take a

Re: [Discuss] FLIP-362: Support minimum resource limitation

2023-10-02 Thread David Morávek
H Xiangyui, The sentiment of the FLIP makes sense, but I keep wondering whether this is the best way to think about the problem. I assume that "interactive session cluster" users always want to keep some spare resources around (up to a configured threshold) to reduce cold start instead of

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

2023-10-02 Thread David Morávek
Hello Yuepeng, The FLIP reads sane; nice work! To re-phrase my understanding: The problem you're trying to solve only exists in complex graphs with different per-vertex parallelism. If the parallelism is set globally (assuming the pipeline has roughly even data skew), the algorithm could make

Re: Flink and Flink shaded dependency

2023-10-02 Thread Jing Ge
Hi Sergey, Thanks for sharing your thoughts. It could somehow help but didn't get to the root of this issue. According to the documentation, Flink shaded is used to provide a single instance of a shaded dependency across sub-modules in Flink repo. Shaded namespaces should be used where shaded

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

2023-10-02 Thread Jing Ge
Hi, I have the same concern as Sergey and would like to make sure the purpose of this discussion is to globally ignore hints without changing any other behaviours, if I am not mistaken. Thanks! Best regards, Jing On Mon, Oct 2, 2023 at 3:40 PM Sergey Nuyanzin wrote: > Hi Bonnie, > > I think

[jira] [Created] (FLINK-33177) Memory leak in MockStreamingRuntimeContext

2023-10-02 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-33177: --- Summary: Memory leak in MockStreamingRuntimeContext Key: FLINK-33177 URL: https://issues.apache.org/jira/browse/FLINK-33177 Project: Flink Issue Type: Bug

Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints

2023-10-02 Thread Sergey Nuyanzin
Hi Bonnie, I think changing it to something like .enabled could lead to misunderstanding for instance when we set this flag to false what should it mean? 1. Just ignore hints and execute a query like the same query however with removed hints 2. Fail on validation because hints are disabled 3.

[jira] [Created] (FLINK-33176) Kinesis source throws NullPointerException in Table API on ignored parsing errors

2023-10-02 Thread Aleksandr Pilipenko (Jira)
Aleksandr Pilipenko created FLINK-33176: --- Summary: Kinesis source throws NullPointerException in Table API on ignored parsing errors Key: FLINK-33176 URL: https://issues.apache.org/jira/browse/FLINK-33176

[jira] [Created] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests

2023-10-02 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-33175: -- Summary: Nightly builds from S3 are not available for download, breaking all connector tests Key: FLINK-33175 URL: https://issues.apache.org/jira/browse/FLINK-33175