Re: Adjusted frame length exceeds 2147483647

One other pointer: Martijn mentioned in FLINK-24923 [1] tools like Nessus could generate traffic while scanning for ports. It's just the size of the request that is suspicious. [1] https://issues.apache.org/jira/browse/FLINK-24923 On Thu, Mar 17, 2022 at 5:29 PM Ori Popowski wrote: > This issue

Re: Weird Flink Kafka source watermark behavior

Thanks Dongwon! Wow. Yes, I'm using per-partition watermarking [1]. Yes, my # source tasks < # kafka partitions. This should be called out in the docs or the bug should be fixed. On Thu, Mar 17, 2022 at 10:54 PM Dongwon Kim wrote: > Hi Dan, > > Do you use the per-partition watermarking expla

Re: Weird Flink Kafka source watermark behavior

Hi Dan, Do you use the per-partition watermarking explained in [1]? I've also experienced a similar problem when running backfill jobs specifically when # source tasks < # kafka partitions. - When # source tasks = # kafka partitions, the backfill job works as expected. - When # source tasks < # ka

Clarifying ProcessFunction.onTimer and watermark behavior

Hi. This Flink page says the following: “With event-time timers, the onTimer(...) method is called when the current watermark is advanced up to or beyond the timestamp of the timer” The do

Re: Weird Flink Kafka source watermark behavior

I'm following the example from this section: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#watermark-strategies-and-the-kafka-connector On Thu, Mar 17, 2022 at 10:26 PM Dan Hill wrote: > Other points > - I'm using the kafka timestamp a

Kafka source with multiple partitions loses data during savepoint recovery

Hi, I'm seeing an odd behavior for Kafka source where some records are dropped during recovery. My test set up is: Kafka source topic -> pass through flink job -> Kafka sink topic There are 10 partitions in the source & sink topics. Test Steps * Start the flink job, send 5 records (first batch)

Re: Weird Flink Kafka source watermark behavior

Other points - I'm using the kafka timestamp as event time. - The same issue happens even if I use an idle watermark. On Thu, Mar 17, 2022 at 10:17 PM Dan Hill wrote: > There are 12 Kafka partitions (to keep the structure similar to other low > traffic environments). > > On Thu, Mar 17, 2022 at

Re: Weird Flink Kafka source watermark behavior

There are 12 Kafka partitions (to keep the structure similar to other low traffic environments). On Thu, Mar 17, 2022 at 10:13 PM Dan Hill wrote: > Hi. > > I'm running a backfill from a kafka topic with very few records spread > across a few days. I'm seeing a case where the records coming from

Weird Flink Kafka source watermark behavior

Hi. I'm running a backfill from a kafka topic with very few records spread across a few days. I'm seeing a case where the records coming from a kafka source have a watermark that's more recent (by hours) than the event time. I haven't seen this before when running this. This violates what I'd as

[File partition Flink]

Hi team, I have some questions about the format when I process the filesIn-progress / Pending: part--.inprogress.uidFinished: part--Can you explain more about the partFileIndex since the format of the files is quite weird. It produces two files (I wonder it related to the parallelism which we have

Re: onTimer() of a KeyedProcessFunction stops getting triggered after a while

Hi, can you share your code so we can check whether it is written correctly. > On 18 Mar 2022, at 7:54 AM, Binil Benjamin wrote: > > Hi, > > We have a class that extends KeyedProcessFunction and overrides onTimer() > method. During processElement(), we register a timer callback using > cont

onTimer() of a KeyedProcessFunction stops getting triggered after a while

Hi, We have a class that extends KeyedProcessFunction and overrides onTimer() method. During processElement(), we register a timer callback using context.timerService().registerProcessingTimeTimer(). For a while, we see that the onTimer() method is getting called back and everything works as expec

Re: Flink UI - Operator Chaining - broken with "Records Sent"

This is on Flink v1.12.3. On Thu, Mar 17, 2022 at 3:16 PM Dan Hill wrote: > Hi. I have an operator that Flink chained together with some side output > operators. Even though the main output of the operator goes to another > operator, the "Records Sent" metric is still zero. I'd expect it to b

Flink UI - Operator Chaining - broken with "Records Sent"

Hi. I have an operator that Flink chained together with some side output operators. Even though the main output of the operator goes to another operator, the "Records Sent" metric is still zero. I'd expect it to be the number of records of the main output (not the side sink operators). Is this

Re: Flink Kinesis checkpoint failures long running service

Hi Hunag, It says checkpoint *Expired *with following Log: 2022-03-16 03:03:22,641 INFO org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher - Shutting down the shard consumer threads of subtask 13 ... 2022-03-16 03:03:22,641 WARN *org.apache.hadoop.fs.s3a.S3ABlockOutputStre

Correct way to cleanly shut down StateFun Harness in test code

Hi all, I’m using org.apache.flink.statefun.flink.harness.Harness in some unit test code, where I control the sources so that they are finite. This is similar to what I found at https://stackoverflow.com/questions/61939681/is-it-possible-to-write-a-unit-test-which-terminates-using-flink-statefu

Re: how to set kafka sink ssl properties

Hi Matthias, It should be probably be like this: Properties SinkkafkaProps = new Properties(); SinkkafkaProps.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, outputBrokers); SinkkafkaProps.setProperty("ssl.truststore.type", trustStoreType); SinkkafkaProps.setProperty("ssl.truststore.locati

Re: [External] Re: Potential Bug with Date Serialization for Table Stream

Hi Martijn, We are using 1.11.6. Thank you for the help. On Thu, Mar 17, 2022 at 1:37 AM Martijn Visser wrote: > Hi Tom, > > Which version of Flink are you using? > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > > On Wed, 16 Mar 2022 at 23:59, Tom Thornton wrote:

RE: Interval join operator is not forwarding watermarks correctly

Hi Dawid, Thanks for the update, I also managed to work around it by adding another watermark assignment operator between the join and the window. I’ll have to see if it’s possible to assign watermarks at the source, but even if it is, I worry that the different partitions created by all my key

Re: how to set kafka sink ssl properties

Could you share more details on what's not working? Is the ssl.trustore.location accessible from the Flink nodes? Matthias On Thu, Mar 17, 2022 at 4:00 PM HG wrote: > Hi all, > I am probably not the smartest but I cannot find how to set ssl-properties > for a Kafka Sink. > My assumption was tha

Re: Adjusted frame length exceeds 2147483647

This issue did not repeat, so it may be a network issue On Thu, Mar 17, 2022 at 6:12 PM Matthias Pohl wrote: > Hi Ori, > that looks odd. The message seems to exceed the maximum size of 2147483647 > bytes (2GB). I couldn't find anything similar in the ML or in Jira that > supports a bug in Flink.

Re: Adjusted frame length exceeds 2147483647

Hi Ori, that looks odd. The message seems to exceed the maximum size of 2147483647 bytes (2GB). I couldn't find anything similar in the ML or in Jira that supports a bug in Flink. Could it be that there was some network issue? Matthias On Tue, Mar 15, 2022 at 6:52 AM Ori Popowski wrote: > I am

Re: Flink Kinesis checkpoint failures long running service

Hi, so the problem is about checkpoints. We need to understand why there are checkpoint failure. Can you provide more logs. We need to check the log to see more details about the first failed checkpoint. > On 17 Mar 2022, at 9:41 AM, Vijayendra Yadav wrote: > >  > Hi Flink Team, > > I am usi

how to set kafka sink ssl properties

Hi all, I am probably not the smartest but I cannot find how to set ssl-properties for a Kafka Sink. My assumption was that it would be just like the Kafka Consumer KafkaSource source = KafkaSource.builder() .setProperties(kafkaProps) .setProperty("ssl.truststore.type", trustStoreT

ClassCastException occurs when Flink DataStream sends a message to a remote stateful function

Hello, I’m trying to send a message from DataStream API to a remote Stateful Function (Java SDK). The DataStream job:     public static final FunctionType DEVICE = new FunctionType("com.github.f1xman.era.anomalydetection.device", "DeviceFunction");     public static void main(String[] args) th

Re: Interval join operator is not forwarding watermarks correctly

Hi Alexis, I tried looking into your example. First of all, so far, I've spent only a limited time looking at the WatermarkGenerator, and I have not thoroughly understood how it works. I'd discourage assigning watermarks anywhere in the middle of your pipeline. This is considered to be an ant

Re: Potential Bug with Date Serialization for Table Stream

Hi Tom, Which version of Flink are you using? Best regards, Martijn Visser https://twitter.com/MartijnVisser82 On Wed, 16 Mar 2022 at 23:59, Tom Thornton wrote: > Per the docs , I'm > hoping to confirm whether or not an error we are seei