Flink SQL build-in function questions.

hi all, I met two problems when I use FlinkSQL. 1. Is there any plan to support bit operation functions ? Currently there is some jira mentioned about this, https://issues.apache.org/jira/browse/FLINK-14990 , https://issues.apache.org/jira/browse/FLINK-12451 But It seems that it hasn't been u

unaligned checkpoint metric

hi all, how do I interpret the checkpointAlignmentTime value when i am using unaligned checkpointing since there is no alignment? thanks, kelvin

Re: Re: Dependency injection for TypeSerializer?

Thanks for the feedback! @Seth Your suggestion should work, I yet have to try it out. However relying on undocumented behavior (return CompatibleAsIs and its serializer will never be used) would make me hesitant to adopt it as permanent solution. @Arvid There is no issue with constructing the

Re: How to express the datatype of sparksql collect_list(named_struct(...))inflinksql?

There are multiple ways of having a more generic UDF. I will use pseudo code here: // supports any input def eval(@DataTypeHint(inputGroup = ANY) Object o): String = { } // or you use no annotations at all and simply define a strategy // default input strategy is wildcard def eval(Map[Row, Int

Re: Upgrade from 1.13.1 to 1.13.2/1.13.3 failing

Thank you for your response. On Tue, Nov 9, 2021 at 6:55 AM Dawid Wysakowicz wrote: > Hey Sweta, > > Sorry I did not get back to you earlier. > > Could you explain how do you do the upgrade? Do you try to upgrade your > cluster through HA services (e.g. zookeeper)? Meaning you bring down the > 1

Re: Kafka Source Recovery Behavior

Hi all, Any update on this? Best, Mason On Sat, Oct 30, 2021 at 5:56 AM Arvid Heise wrote: > This seems to be a valid concern but I'm not deep enough to clearly say > that this is indeed a bug. @renqschn could you > please double-check? > > On Thu, Oct 28, 2021 at 8:39 PM Mason Chen wrote: >

Flink docker on k8s job submission timeout

Hello all, We are trying to run a Flink job in standalone mode using the official docker image on k8s. As per this documentation we have created our custom docker i

Re: select records using JDBC with parameters

Thanks alot it was really related to different versions. I have one more question about this solution the select statement returns list of results i see that when retrieving data we activate row mapper which handles only one row at a time and return PCollection of that row do i have a way to aggreg

Re: Dependency injection for TypeSerializer?

Yes I did, thanks for sending it back :) Copying my previous reply for the ML: Hey Thomas, > > You are correct that there is no way to inject dynamic information into > the TypeSerializer configured from the TypeSerializerSnapshot, but that > should not be a problem for your use case. > > The type

Re: Datastream processing on AWS with Python model

Hi Des, Regarding kinesis datastream source: currently it still hasn't supported kinesis source natively in PyFlink DataStream API, however, you could use the Kinesis Table API & SQL connectors [1] and then convert the Table to DataStream [2] if you want to work with PyFlink DataStream API. Regar

Datastream processing on AWS with Python model

Hi all, About our use case - We have many clients connected via websockets through api gateway on AWS, these clients submit events of various types periodically, each event contains a session_id (generated by the client), the session ends when there's no activity for a specified duration of time.

Re: Access to GlobalJobParameters From DynamicTableSourceFactory

Well, the use case is rather simple, just pass a JVM parameter to the Table Source's underlying code and the property should not be exposed to Table Definition. Like something that end user should not change but its driven via application properties. For example you are running SQL that should use

Re: JVM cluster not firing event time window

Hi! Is the number of partitions of your Kafka topic smaller than the number of parallelisms of the job? If yes some parallelism will be idle and will not emit watermarks unless you set idleness for them. See [1]. I guess the original behavior of Flink 1.12 is not the expected behavior but I don't

Re: Dependency injection for TypeSerializer?

Hi Thomas, Could you add a sketch of your preferred solution? From what I gathered, you have all the information available in your main (probably misunderstood that), so what's keeping you from adding the TypeSerializer as a field to your UDF? On Tue, Nov 9, 2021 at 11:42 AM Krzysztof Chmielewski

Re: Pyflink PyPi build - scala 2.12 compatibility

Thank you for the clarification. This was very helpful! Kind regards Kamil On Wed, 10 Nov 2021, 02:26 Dian Fu, wrote: > Hi Kamil, > > You are right that it comes with JAR packages of scala 2.11 in the PyFlink > package as it has to select one version of JARs to bundle, either 2.11 or > 2.12. Wh

Re: how to expose the current in-flight async i/o requests as metrics?

Hi Dongwon, Thanks for sharing the logs and the metrics screenshots with us. Unfortunately, I think we need more information to further isolate the problem therefore I have a couple of suggestions. 1. Since you already set up PromQL can you also share the JVM memory statics i.e. DirectMemory cons

How to specify slot task sharing group for a task manager?

Hi :) I am trying the fine-grained resource management feature in Flink 1.14, hoping it can enable assigning certain operators to certain TaskManagers. The sample code in https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/finegrained_resource/ -shows how to define the