[jira] [Created] (FLINK-21643) JDBC sink should be able to execute statements on multiple tables
Maciej Obuchowski created FLINK-21643: - Summary: JDBC sink should be able to execute statements on multiple tables Key: FLINK-21643 URL: https://issues.apache.org/jira/browse/FLINK-21643 Project: Flink Issue Type: New Feature Components: Connectors / JDBC Affects Versions: 1.12.2 Reporter: Maciej Obuchowski Currently datastream JDBC sink supports outputting data only to one table - by having to provide SQL template, from which SimpleBatchStatementExecutor creates PreparedStatement. Creating multiple sinks, each of which writes data to one table is impractical for moderate to large number of tables - relational databases don't usually tolerate large number of connections. I propose adding DynamicBatchStatementExecutor, which will additionally require 1) provided mechanism to create SQL statements based on given object 2) cache for prepared statements 3) mechanism for determining which statement should be used for given object -- This message was sent by Atlassian Jira (v8.3.4#803005)
QUESTION: Local Dev Setup
Hi, I've been making some modifications to the flink source code and a connector. Now I would like to test my code by building a job with it and running the job against my modified flink source. There are multiple options, I guess, how to achieve that. So, I wanted to ask here about your setup / best practices / tips on how to work with, develop and test custom flink builds. How does your setup and your usual workflow look like? Best regards and thanks Jan
[jira] [Created] (FLINK-21642) RequestReplyFunction recovery fails with a remote SDK
Igal Shilman created FLINK-21642: Summary: RequestReplyFunction recovery fails with a remote SDK Key: FLINK-21642 URL: https://issues.apache.org/jira/browse/FLINK-21642 Project: Flink Issue Type: Bug Components: Stateful Functions Reporter: Igal Shilman While extending our smoke e2e test to use the remote SDKS I've stumbled upon a bug in the RequestReplyFunction. We get a unknown state exception after recovery. The exact scenario that trigger that bug is: # There was request in flight. # A failure occurs that causes the job to restart. # On restore, we start with no managed state # But we try to re-send to the SDK exactly the same ToFunction message. # That ToFunction contains state definitions from the previous attempt. (before the failure) # The SDK processes this message normally (it has all the state definitions that it knows) # The SDK responds with a state mutation. # The PersistedRemoteFunctionValues fails with unknown state. We need to treat the ToFunction messages as a retryBatch, instead of sending it as-is. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21641) Support 'useModules' and 'listModules' in PyFlink
Jane Chan created FLINK-21641: - Summary: Support 'useModules' and 'listModules' in PyFlink Key: FLINK-21641 URL: https://issues.apache.org/jira/browse/FLINK-21641 Project: Flink Issue Type: Sub-task Components: API / Python Affects Versions: 1.13.0 Reporter: Jane Chan -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [ANNOUCE][DISCUSS] Roadmap Update for Website
Thanks a lot for updating the roadmap! I really like the new Feature Radar, because it makes some assumptions floating around on JIRA and the lists properly documented. I guess the radar is missing the Reactive Mode as an MVP feature. Otherwise, I'm +1 on publishing this soon! On Tue, Mar 2, 2021 at 3:38 PM Stephan Ewen wrote: > Hi all! > > The roadmap on the Flink website is quite outdated: > https://flink.apache.org/roadmap.html > > I drafted an update to the roadmap that reflects the currently ongoing > bigger threads. > Not every detail is mentioned there, because this roadmap should give users > a high-level view where the project is going. > > There is also a new "Feature Radar" to help users understand in which stage > of maturity and support individual features are. > > Now, I am sure this will cause some discussion about which feature > should be on which level of maturity. This is my initial proposal which I > checked with some committers, but of course, there may be different > opinions. In that case, please bring this up in this discussion thread and > we find consensus together. > > > https://docs.google.com/document/d/1g6T72-8PHTsfhZq8GWWzG8F3PmD-dMuFSzg2AozXNxk/edit?usp=sharing > > The raw figure for the feature radar is: > > https://github.com/StephanEwen/flink-web/blob/feature_radar/img/flink_feature_radar.svg > > If someone has some graphics skills to make this look more pretty, help is > greatly welcome! > (Only request would be to keep a format that many people (and open tools) > can edit, so maintenance remains easy). > > Looking forward to hearing what you think! > > Best, > Stephan >
Re: [DISCUSSION] Introduce a separated memory pool for the TM merge shuffle
Thanks Guowei, for the proposal. As discussed offline already, I think this sounds good. One thought is that 16m sounds very small for a default read buffer pool. How risky do you think it is to increase this to 32m or 64m? Best, Stephan On Fri, Mar 5, 2021 at 4:32 AM Guowei Ma wrote: > Hi, all > > > In the Flink 1.12 we introduce the TM merge shuffle. But the > out-of-the-box experience of using TM merge shuffle is not very good. The > main reason is that the default configuration always makes users encounter > OOM [1]. So we hope to introduce a managed memory pool for TM merge shuffle > to avoid the problem. > Goals > >1. Don't affect the streaming and pipelined-shuffle-only batch setups. >2. Don't mix memory with different life cycle in the same pool. E.g., >write buffers needed by running tasks and read buffer needed even after >tasks being finished. >3. User can use the TM merge shuffle with default memory >configurations. (May need further tunings for performance optimization, but >should not fail with the default configurations.) > > Proposal > >1. Introduce a configuration `taskmanager.memory.network.batch-read` >to specify the size of this memory pool. The default value is 16m. >2. Allocate the pool lazily. It means that the memory pool would be >allocated when the TM merge shuffle is used at the first time. >3. This pool size will not be add up to the TM's total memory size, >but will be considered part of >`taskmanager.memory.framework.off-heap.size`. We need to check that the >pool size is not larger than the framework off-heap size, if TM merge >shuffle is enabled. > > > In this default configuration, the allocation of the memory pool is almost > impossible to fail. Currently the default framework’s off-heap memory is > 128m, which is mainly used by Netty. But after we introduced zero copy, the > usage of it has been reduced, and you can refer to the detailed data [2]. > Known Limitation > Usability for increasing the memory pool size > > In addition to increasing `taskmanager.memory.network.batch-read`, the > user may also need to adjust `taskmanager.memory.framework.off-heap.size` > at the same time. It also means that once the user forgets this, it is > likely to fail the check when allocating the memory pool. > > > So in the following two situations, we will still prompt the user to > increase the size of `framework.off-heap.size`. > >1. `taskmanager.memory.network.batch-read` is bigger than >`taskmanager.memory.framework.off-heap.size` >2. Allocating the pool encounters the OOM. > > > An alternative is that when the user adjusts the size of the memory pool, > the system automatically adjusts it. But we are not entierly sure about > this, given its implicity and complicating the memory configurations. > Potential memory waste > > In the first step, the memory pool will not be released once allocated. This > means in the first step, even if there is no subsequent batch job, the > pooled memory cannot be used by other consumers. > > > We are not releasing the pool in the first step due to the concern that > frequently allocating/deallocating the entire pool may increase the GC > pressue. Investitations on how to dynamically release the pool when it's no > longer needed is considered a future follow-up. > > > Looking forward to your feedback. > > > > [1] https://issues.apache.org/jira/browse/FLINK-20740 > > [2] https://github.com/apache/flink/pull/7368. > Best, > Guowei >
