Re: Error while deploying from snapshot after adding new column in existing table

Thanks Martijn! One more question, Can we achieve(schema evolution) using DataStream APIs? In flink documentation, I found [2] "The process of migrating state to adapt to changed schemas happens automatically, and independently for each state. This process is performed internally by Flink by firs

Re: How to handle java.lang.OutOfMemoryError: Metaspace

Given the numbers above from JCMD. You think I should be ok with 2GB metaspace? That was captured while all jobs where running on the cluster for that 1 node. I set it to 2GB. But none of the above numbers indicated max 2GB metaspace. On Mon., Dec. 27, 2021, 10:47 a.m. John Smith, wrote: > Ok

Re: Unable to update logback configuration in Flink Native Kubernetes

I've faced the same issue before. I figured out that there is an internal configuration `$internal.deployment.config-dir` (code ) wh

Re: TypeInformation | Flink

Hi David, Yes, I already mentioned that I am a newbie to Flink and Scala. I am making progress as the day progresses. I have modified my question again. But I am not sure how to use it. Could you please correct it? or add something if I missed something? On Wed, Dec 29, 2021 at 10:53 PM David Mor

Re: TypeInformation | Flink

Hi Siddhesh, You can not change the method signature when you're implementing an interface. I'm not really sure this belongs to the ML anymore as this is getting more into Scala / Java fundamentals. There are some great learning resources online for Scala [1], I'd recommend starting from there. A

Re: TypeInformation | Flink

I have modified my question based on Dominik's inputs. Can somebody help to take it forward? Thanks, Siddhesh On Wed, Dec 29, 2021 at 3:32 PM David Morávek wrote: > Please always try to include user@f.a.o in your reply, so other can > participate in the discussion and learn from your findings.

Unable to update logback configuration in Flink Native Kubernetes

Hi I have created a Flink Native Kubernetes (1.14.2) cluster which is successful. I am trying to update the logback configuration for which I am using the configmap exposed by Flink Native Kubernetes. Flink Native Kubernetes is creating this configmap during the start of the cluster and deleting i

Re: Mapstate got wrong UK when restored.

The problem is that you're not actually using the underlying state during runtime, but instead you're simply using a java map abstraction. This property ("Map state") is simply bound to the UDF lifecycle and doesn't share the semantics of the keyed state. You should be using the "MapState" propert

Re: Flink and Datadog metrics question

Hi Adrian, for accessing metrics in your UDFs (user defined functions), you need access to the runtime context, unless you're using lower level APIs that directly extend stream operator. There are "rich" versions of most of the higher level functions (map, flatmap, group combine, ...). Can you el

Re: TypeInformation | Flink

Okay, David. Let me try it and will let you know. On Wed, Dec 29, 2021 at 3:32 PM David Morávek wrote: > Please always try to include user@f.a.o in your reply, so other can > participate in the discussion and learn from your findings. > > I think Dominik has already given you pretty good hint. T

Flink and Datadog metrics question

Hi, I'm using flink version 1.12.1, batch jobs in an application cluster and Datadog to expose metrics. I want to know if someone know how to send metrics from non-extended RichMapFunction class. At this moment we can expose all the metrics from each operator that extends from RichMapFunction bu

Re: Anyone trying to adopt Scotty on the recent Flink versions?

Hi David, Scotty's approach to sliding windows seems really interesting ;) I also agree that the idea is very interesting. It will be very effective for our use case as we create a sliding window of size 10 minutes that slides by 1 minutes for every road of the country. Looking at the code [1],

Re: Mapstate got wrong UK when restored.

Hi David, Thanks for you reply. Yes, for keyed state, every state is referenced by a particular key, but I would guess it is a flink sdk issue, I mean, the keyed state maybe saved as (key, keyed state), as for my situation, it is (key, mapstate(UK,UV)), I think the key of this pair is not easy to

Re: Re: Read parquet data from S3 with Flink 1.12

I've answered in other thread [1]. Please keep the conversation focused there. [1] https://lists.apache.org/thread/7cqqzno3lz75qw9yxprgg45q6voonsbq Best, D. On Tue, Dec 28, 2021 at 4:00 PM Rohan Kumar wrote: > Hi Alexandre, I am also facing the same issue. Please let us know if you > are able

Re: Unable to read S3 data using the filesystem connector

Hi Rohan, setting this up is currently not really straightforward :( I think we need to improve on this. For supporting the s3 filesystem, you did it right by placing s3 jars into the plugins directory. Please note, that these are loaded in a separate class loader and also contain a shaded versio

Re: Error while deploying from snapshot after adding new column in existing table

Hi Shamit, Adding columns means that you're trying to perform schema evolution, which isn't yet supported by Flink per the documentation [1] "Savepoints are only supported if both the query and the Flink version remain constant" Best regards, Martijn [1] https://nightlies.apache.org/flink/flink

Re: TypeInformation | Flink

Please always try to include user@f.a.o in your reply, so other can participate in the discussion and learn from your findings. I think Dominik has already given you pretty good hint. The JSON parsing in this case is not any different as with any other java application (with jackson / gson / ...).

Re: TypeInformation | Flink

Hi David, Thanks for the clarification. I will check the link you shared. Also, as mentioned by Dominik, can you help me with the process functions. How can I use it for my use case? Thanks, Siddhesh On Wed, Dec 29, 2021 at 3:22 PM Siddhesh Kalgaonkar < kalgaonkarsiddh...@gmail.com> wrote: > H

Re: Unable to read S3 data using the filesystem connector

I am running flink 1.14.2 Thanks On Wed, 29 Dec 2021 at 13:18, Rohan Kumar wrote: > Hello, > > I tried to read parquet data in S3 using the filesystem connector but got > the below error. The jobmanger is not starting. > I tried the standalone-job in docker. > I have already included flink-s3-f

Re: Mapstate got wrong UK when restored.

Hi Josh, it's important bit to understand is that the MapState (or any other keyed state) is scoped per *key* [1]. You can think about it in a way, that for each key you have a separate "map" that backs it. This is the important concept behind distributed stream processing, that allows you to para

Re: TypeInformation | Flink

Hi Siddhesh, it seems that the question is already being answered in the SO thread, so let's keep the discussion focused there. Looking at the original question, I think it's important to understand, that the TypeInformation is not meant to be used for "runtime" matching, but to address the type

Re: Anyone trying to adopt Scotty on the recent Flink versions?

Hi Dongwon, Scotty's approach to sliding windows seems really interesting ;) Looking at the code [1], it seems to be no longer maintained. It's both compiled and tested against Flink 1.8 so I wouldn't really expect it to be compatible with 1.14.x :( [1] https://github.com/TU-Berlin-DIMA/scotty-wi

Re: Remove stackTrace from error response

Hi Noa, There is currently no way to do this without making changes to the code. Please note that there are also endpoints for explicitly retrieving the exception history of a particular job. Flink REST API is not really meant to be "secure" in a way that you can make it accessible to the public n

Re: CVE-2021-44228 - Log4j2 vulnerability

Please follow the above mentioned ML thread for more details. Please note that this is a REGULAR release that is not motivated by the log4j CVE, so the stability of the release is the more important factor then having it out as soon as possible. D. On Mon, Dec 27, 2021 at 6:33 AM narasimha wrote