AIFlow flink java job plugin

Hello, [Apologies if this group does not answer questions related to AIFlow project and happy to learn if there are other email handles I need to send my questions to] I am new to AIFlow and exploring some demo projects for a simple workflow I want to try with two flink jobs, a batch (bounded

Re: Best practice for adding support for Kafka variants

t 10:04 AM Chesnay Schepler > wrote: > >> The FLIP-27 were primarily aimed at the DataStream API; the integration >> into the SQL/Table APIs will happen at a later date. >> >> On 6/1/2021 5:59 PM, deepthi Sridharan wrote: >> >> Thank you, Roman. I should

Re: Best practice for adding support for Kafka variants

; > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > [2] > https://issues.apache.org/jira/browse/FLINK-18323 > [3] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API > > Regards, >

Best practice for adding support for Kafka variants

Hi, We have an internal version of Open source Kafka consumer and producer that we use and are working on adding that as a source and sink for flink. It seems like the easiest way to add the consumer as source would be to override the FlinkKafkaConsumer class's createFetcher

Re: Compression with rocksdb backed state

t; blackbox for Flink and Flink's compression flag has no impact. So we > fully rely what RocksDB offers. > 4. Correct > > I hope this helps. > > Regards, > Timo > > > [1] > https://flink.apache.org/features/2018/01/30/incremental-checkpointing.html > > > On 07.04.2

Compression with rocksdb backed state

I am trying to understand this section on compression of checkpoints which has me a bit confused https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/large_state_tuning.html#compression Could you please confirm if my understanding is correct: 1. Compression is disabled by default for

Re: IO benchmarking

he/flink/runtime/checkpoint/CheckpointCoordinator.java#L1493 > [4] > https://github.com/apache/flink/blob/adaaed426c2e637b8e5ffa3f0d051326038d30aa/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L83 > > On Tue, Mar 30, 2021 at 8:37 PM

IO benchmarking

Hi, I am trying to set up some benchmarking with a couple of IO options for saving checkpoints and have a couple of questions : 1. Does flink come with any IO benchmarking tools? I couldn't find any. I was hoping to use those to derive some insights about the storage performance and extrapolate

Multiple JobManager HA set up for Standalone Kubernetes

I am trying to figure out the right architecture for running Standalone Kubernetes with Job manager HA. The documentation for running HA seems to always suggest that there needs to be multiple job managers, but there