Re: [ANNOUNCE] Apache Flink 1.18.0 released

Great work, thanks everyone! Best, Ron Alexander Fedulov 于2023年10月27日周五 04:00写道： > Great work, thanks everyone! > > Best, > Alexander > > On Thu, 26 Oct 2023 at 21:15, Martijn Visser > wrote: > > > Thank you all who have contributed! > > > > Op do 26 okt 2023 om 18:41 schreef Feng Jin > > >

Re: [ANNOUNCE] Apache Flink 1.18.0 released

Great work, thanks everyone! Best, Ron Alexander Fedulov 于2023年10月27日周五 04:00写道： > Great work, thanks everyone! > > Best, > Alexander > > On Thu, 26 Oct 2023 at 21:15, Martijn Visser > wrote: > > > Thank you all who have contributed! > > > > Op do 26 okt 2023 om 18:41 schreef Feng Jin > > >

Re: Flink SQL runtime

Hi, What SQL Runtime are you referring to? Why do you need to get it? Best, Ron Enric Ott <243816...@qq.com> 于2023年10月12日周四 14:26写道： > Hi,Team: > Is there any approach to get flink sql runtime via api ? > Any help would be appreciated. >

Re: Possible bug when converting table to upsert changelog stream

Hi, Charles It is not a bug, this is because the primary keys provided by sink are not exactly match input changeLogUpsertKeys, so fallback to before and after mode, you can see [1] for more detail. [1]

Re: Maven plugin to detect issues in Flink projects

make this a database of known issues detectable during > build/packaging. It’ll be a collaborative effort! > > > > Thanks > > Emre > > > > > > *From: *liu ron > *Date: *Monday, 18 September 2023 at 03:10 > *To: *"user@flink.apache.org" &g

Re: Urgent: Mitigating Slow Consumer Impact and Seeking Open-SourceSolutions in Apache Kafka Consumers

Hi, Karthick It looks like a data skewing problem, and I think one of the easiest and most efficient ways for this issue is to increase the number of Partitions and see how it works first, like try expanding by 100 first. Best, Ron Karthick 于2023年9月17日周日 17:03写道： > Thanks Wei Chen, Giannis

Re: Maven plugin to detect issues in Flink projects

Hi, Emre Thanks for driving this proposal. It looks cool. Intuitively though, I don't really see what type of compatibility issues it's trying to solve, can you explain in a bit more detail? Is it solving compatibility issues within the Flink project itself, or compatibility issues with the Flink

Re: Flink upgrade 1.14.6 to 1.15.4 -> StackOverflowError

Hi, Alexey Have you tried other versions like 1.15.3 or 1.16.1? Best, Ron Alexey Trenikhun 于2023年9月15日周五 04:24写道： > Hello, > After upgrading Flink 1.14.6 to 1.15.4 (Kubernetes, Application mode) job > started to failed due to StackOverflowError. The application uses > okhttp3 4.9.2, but it is

Re: Reading parquet files using Flink

Hi, Are you using the DataStream API to read parquet file? Why not use Flink SQL to read the it? The ParquetRowInputFormat has been removed, you can use ParquetColumnarRowInputFormat in 1.17.1. Best, Ron Hou, Lijuan via user 于2023年9月12日周二 05:49写道： > Hi team, > > > > Is there any defined way

Re: Order of Execution in KeyedBroadcastProcessFunction

Hi, The KeyedBroadcastProcessFunction#processBroadcastElement is called in outer CoBroadcastWithKeyedOperator[1], CoBroadcastWithKeyedOperator is a two-input operator. As David says, this operator is run in a single thread, so the processBroadcastElement method will run to completion before any

Re: Watermarks

Hi, Sid For the second question, I think it is not needed. Best, Ron Feng Jin 于2023年9月9日周六 21:19写道： > hi Sid > > > 1. You can customize KafkaDeserializationSchema[1], in the `deserialize` > method, you can obtain the Kafka event time. > > 2. I don't think it's necessary to explicitly mention

Re: using CheckpointedFunction on a keyed state

Hi Krzysztof Can you give the original code in initializeState method and the corresponding exception stack trace? It looks a little interesting. Best, Ron Krzysztof Chmielewski 于2023年9月9日周六 07:12写道： > My apologies Mattthias, > you are right. The issue was that I was trying to access

Re: Problem when testing table API in local

Hi, Why don't you add the flink-table-planner module as dependency directly for test. Best, Ron Oscar Perez via user 于2023年9月8日周五 21:44写道： > quick update, > > after adding flink-clients it *worked *the first time I ran the test but > then the second time got the same error. Looks like a race

Re: Serialize and Parse ResolvedExpression

Hi, Xianxun Do you mean the unix_timestamp() is parsed to the time when the query is compiled in streaming mode? Best, Ron Xianxun Ye 于2023年9月7日周四 18:19写道： > Hi Team, > > I want to Serialize the ResolvedExpression to String or byte[] and transmit > it into LookupFunction, and parse it back to

Re: Async IO metrics for tps

Hi, What's your question? Best Ron patricia lee 于2023年9月7日周四 14:29写道： > Hi flink users, > > I used Async IO (RichAsyncFunction) for sending 100 txns to a 3rd party. > > I check the runtimeContex that it has metric of numRecordsSent, we wanted > to expose this metric to our prometheus server

Re: How to read flinkSQL job state

Hi, Yifan Flink SQL & Table API currently doesn't support reading the state directly. Best, Ron Yifan He via user 于2023年9月6日周三 13:11写道： > Hi team, > > We are investigating why the checkpoint size of our FlinkSQL jobs keeps > growing and we want to look into the checkpoint file to know what is

Re: backpressured metrics doesnt work

Hi, Kenan I think you need to provide more context, maybe help to find the root cause. Best, Ron Kenan Kılıçtepe 于2023年9月4日周一 21:49写道： > Hi, > > Any idea why backpressured metrics are not working and how I can fix it? > > [image: image.png] > > Thanks > Kenan > >

Re: Send data asynchronously to a 3rd party via SinkFunction

Hi, patricia If you want to use SinkFunction, maybe you should use `RichSinkFunction`[1], you can close the resources in close method. [1]

Re: Question regarding asyncIO timeout

Hi, Leon > Besides that, Do you know if the async timeout is actually a global timeout? meaning it accounts for the time of each attempt call plus any interval time in between. Yes, the timeout is total timeout, you can see [1][2] for more detail. [1]

Re: Access to collector in the process function

Hi, Oscar The collector object is created in [1] open method, so we can't get it in ProcessFunciton open method. One possible way is you change the source code, place it in ProcessFunction#Context. [1]

Re: Job graph

Hi, Nikolaos As far as I know, JobGraph is a relatively low-level concept, and currently we don't expose it directly to users, and don't provide a direct Restful API to get it from JobManager. Why do you need to get JobGraph and what is your real need? Best, Ron Nikolaos Paraskakis

Re: 退订

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2]

Re: Table API table2datastream (toChangelogStream)

Hi, Ravi I have deep dive into the source code[1], the parallelism of the Sink operator is consistent with its inputs, so I suggest you check the parallelism of the upstream operators. [1]

Re: Usecase advise for Apache Flink Table API

Hi, Nirmal Flink SQL is standard ANSI SQL and extends upon it. Flink SQL provides rich Join and Aggregate syntax including Regular Streaming Join, Interval Join, Temporal Join, Lookup Join[2], Window Join[3], unbounded group aggregate[4] and window aggregate[5], and so on. Theoretically, it can

Re: How to use pipeline.jobvertex-parallelism-overrides property.

Hi, Krzysztof As stated in the description section, this option is used to override the parallelism of a JobVertex, where the key is JobVertex id, you can see [1] for double check. A JobVertex may contain more than one operator, so we cannot override the parallelism of a given operator alone. One

Re: Re-start strategy without checkpointing enabled

Hi, Kamal As Hang says, some extra info about job failover strategy for reference in [1] [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/ Best, Ron Hang Ruan 于2023年8月23日周三 22:27写道： > Hi, Kamal. > > If we don't enable checkpointing, the job will

Re: Flink 1.17.2 planned?

Hi, Christian We released 1.17.1 [1] in May, and the main focus of the community is currently on the 1.18 release, so 1.17.2 should be planned for after the 1.18 release! [1] https://flink.apache.org/2023/05/25/apache-flink-1.17.1-release-announcement/ Best, Ron Christian Lorenz via user

Re: TaskManagers Crushing

Hi, Maybe you need to check what changed on the Kafka side at that time. Best, Ron Kenan Kılıçtepe 于2023年8月20日周日 08:51写道： > Hi, > > I have 4 task manager working on 4 servers. > They all crush at the same time without any useful error logs. > Only log I can see is some disconnection from

Re: Splitting in Stream Formats for File Source

Hi, Regarding CSV and AvroParquet stream formats doens't supporting splits, I think some hints may be available from [1]. Personally, I think the main consideration should be the question of how the row format can find a reasonable split point, and how many Splits are appropriate to slice a file

Re: [Issue] Failure on stopping Flink(1.17) application

Hi, I think you can check the client side and JobManager side log to get more info. Best, Ron Dennis Jung 于2023年8月18日周五 10:41写道： > Hello people, > > I'm facing failure when I try to stop running Flink job with REST API > 'jobs/:jobid/stop' > > ``` > ... >

Re: [Question] Good way to monitor data skewness

Hi, Dennis, Although all operators are chained together, each operator metrics is there, you can view the metrcis related to the corresponding operator's input and output records through the UI, as following: [image: image.png] Best, Ron Dennis Jung 于2023年8月16日周三 14:13写道： > Hello people, >

Re: Flink SQL query with window-TVF fails

Hi, Pouria Flink SQL uses the calcite to parse SQL, this is the calcite limitation, the minimum precision it supports is Second [1]. [1] https://github.com/apache/calcite/blob/main/core/src/main/codegen/templates/Parser.jj#L5067 Best, Ron Pouria Pirzadeh 于2023年8月15日周二 08:09写道： > I am trying

Re: Recommendations on using multithreading in flink map functions in java

Hi, Vignesh Flink is a distributed parallel computing framework, each MapFunction is actually a separate thread. If you want more threads to process the data, you can increase the parallelism of the MapFunction without having to use multiple threads in a single MapFunction, which in itself

Re: Question about serialization of java.util classes

Hi, According to the test in [1], I think Flink can recognize Pojo class which contains java List, so I think you can refer to the related Pojo class implementation. [1]

Re: Conversion expects insert-only records but DataStream API record contains: UPDATE_BEFORE

Hi, After deep dive into the source code, I guess you use the StreamTableEnvironment#fromDataStream method, this method only supports the insert-only message. According to your case, I think you should use the StreamTableEnvironment#fromChangelogStream[1], it supports consuming update row. [1]

Re: Local process and Docker have different behavior

Hi, Daniel > 1. The file source appears to not always read all of the files that are in my source directory. There are 3 files that I'm reading in a directory. It will sometimes skip one, seemingly at random. What version of Flink are you using? This looks strange, I think you could check the

Re: Questions related to Autoscaler

Hi, zhanghao Thanks for your reply, it also looks good to me. Chen Zhanghao 于2023年8月11日周五 09:23写道： > Q1: if you use operator to submit a standalone mode job with reactive mode > enabled, KEDA should still work. > > Q2: For Flink versions, 1.17 is recommended, but 1.15 is also okay if you >

Re: Global/Shared objects

Hi, Kamal This is possible, you can use static variable. Best, Ron Hang Ruan 于2023年8月11日周五 13:46写道： > Hi, Kamal. > > Each TaskManager is a JVM process and each task slot is a thread of the > TaskManager. More information see [1]. > The static fields could be shared among subtasks in the same

Re: Flink operator job restart

Hi, lvan You can refer to the five-part that restore the job in [1]. [1] https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.6/docs/operations/upgrade/#2-upgrading-with-existing-flinkdeployments Best, Ron Ethan T Yang 于2023年8月10日周四 12:28写道： > Hi Flink users, > > When

Re: Integarting Apache Flink with Apache Ranger

Hi, Unfortunately, The open-source community doesn't integrate Apache Ranger with Flink, now it is only available in the Cloudera Flink solution[1]. Maybe you can drive this proposal. [1] https://docs.cloudera.com/csa/1.6.1/security/topics/csa-flink-ranger-policies.html Best, Ron arjun s

Re: Easiest way to do a batch outer join

ike > in this case for outer joins or dataset broadcasting). I hope things will > improve for batch scenarios in the future. > > Best, > Flavio > > On Wed, Aug 9, 2023 at 4:55 AM liu ron wrote: > >> Hi, Flavio >> >> IMO, the current DataStream API is not align

Re: About[jobmanager.memory.off-heap.size] not work on Flink(flink-session on yarn)

Hi, wenjiang It does seem a bit odd that it could theoretically work either way. I think you can check the Flink JobManager log to find more information. Best, Ron 傅文江 于2023年7月24日周一 17:37写道： > > When I use ESSink, I find that I need to set Flink’s > jobmanager.memory.off-heap.size to 256MB.

Re: Questions related to Autoscaler

Hi, Lijuan > 1 - It seems for flink job using flink operator to realize autoscaling, the only option to realize autoscaling is to enable the Autoscaler feature, and KEDA won’t work, right? What is KEDA mean? > 2 - I noticed from the document that we need to upgrade to flink version of 1.17 to

Re: Streaming join performance

Hi, David Regarding the N-way join, this feature aims to address the issue of state simplification, it is on the roadmap. Technically there are no limitations, but we'll need some time to find a sensible solution. Best, Ron David Anderson 于2023年8月9日周三 10:38写道： > This join optimization sounds

Re: Easiest way to do a batch outer join

Hi, Flavio IMO, the current DataStream API is not aligned with DataSet in terms of capabilities, I think you can try it with GlobalWindow. Another possible solution is to convert the DataStream to a table[1] first and then try it with a join on the Table API. [1]

Re: Dependency injection framework for flink

Hi, Oscar IMO, Flink as a big data compute engine, its main goal is to provide general-purpose computing power, not as a back-end service, or to solve a specific business problem, so it doesn't need a dependency injection framework, so that's why you didn't find information about it in the Flink

Re: Will all records grouped using keyBy be allocated to a single subtask?

Hi, David. Yes, all records with the same key will be shuffled to a single downstream subtask. Otherwise, the computed results will be wrong. Best, Ron xiangyu feng 于2023年8月4日周五 09:45写道： > Hi David, > > keyBy() is implemented with hash partitioning. If you use the keyBy > function, the

Re: Flink task manager failure scenario

Hi, Kamal According to your description, I think this is related to Flink's fault tolerance mechanism， you can see [1] for more detail. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/ Best, Ron Kamal Mittal via user 于2023年8月4日周五 15:06写道： >

Re: Flink restored from an initially-specified checkpoint

Hi, Fil If we don't specify the ExternalizedCheckpointCleanup, the default checkpoint retention strategy is never retaining after the job terminates, which causes your issue. So I think your configure the ExternalizedCheckpointCleanup to `RETAIN_ON_CANCELLING`. Best, Ron Filip Karnicki

Re: Flink operator task opens threads internally

Hi, Kamal Based on your context, it is okay, but the server socket function must be singleton. Otherwise, there will be port conflict problems. One more question, would you be so kind as to provide the PoC code? Maybe it helps to us share more views. Best, Ron Kamal Mittal via user

Re: Is there any way I can do aggregation on DataStream in Apache Flink and store the Aggregated value?

Hi, Muazim Flink is an incremental computing framework, in streaming mode it considers data to be unbounded, so every piece of data that comes in triggers the computation logic because it doesn't know when the data will end. Based on your description, I understand that you may have a full data

Re: Kafka Disconnection Error

Hi, Kenan I think you maybe can get help from Kafka community. IMO, it is just an info level log, whether it has a real impact? Best, Ron Kenan Kılıçtepe 于2023年8月2日周三 06:04写道： > I got a lot of these disconnection error logs. Why? My flink and kafka > clusters are running in Google Cloud and

Re: No. of task managers vs No. of task slots

Hi, Kamal > How many task managers a job manager can handle? Is there any upper limit also? There is no clear limit to how many TMs a JM can cover, and based on my past experience, it can handle TMs over 1000+, even more. > How to decide no. of task managers, is there any way? I don't think

Re: Unsubscribe

Hi, Please send an email to user-unsubscr...@flink.apache.org in order to be removed from the User mailing list. Best, Ron yunchao li 于2023年7月17日周一 09:14写道： > Unsubscribe -- > >>

Re: Unsubscribe

Hi, wang Please send an email to user-unsubscr...@flink.apache.org in order to be removed from the User mailing list. Best, Ron wang <24248...@163.com> 于2023年7月17日周一 23:10写道： > Unsubscribe >

Re: Unsubscribe

Hi, William Please send an email to user-unsubscr...@flink.apache.org in order to be removed from the User mailing list. Best, Ron William Wang 于2023年7月17日周一 14:59写道： > >

Re: Flink netty connector for TCP source

Hi, Kamal Currently, we don't provide the connector like TCP source in Flink main repo. If you need this connector, you can try to implement it refer to the FLIP-27 source docs [1] https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/sources/ Best, Ron Shammon FY

Re: Metrics are not reported in Python UDF (used inside FlinkSQL) when exception is raised

Hi, Elkhan I have one question, what metrics are not reported? Best, Ron Elkhan Dadashov 于2023年7月28日周五 05:46写道： > Hi Flinkers, > > Wanted to check if anyone else has faced this issue before: > > When Python UDF (which is used inside FlinkSQL) raises an exception, then > metrics get lost and

Re: Flink sql client doesn't work with "partition by" clause

: Flink sql client doesn't work with "partition by" clause > To: liu ron > > > Hi, ron. > > Actually I'm not receiving any exception message when executing the *partition > by* clause in the Flink SQL Client. > > The job does not fail, but it finishes quickly without

Re: Logs of Kafka payload in Table Api connector of Apache Flink

Hi, elakiya If you want to log the deserialize avro record, you should modify the code in [1]. 1. https://github.com/apache/flink/blob/70e635983dc9aa4bd772042f34a04099256a8750/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L81 Best,

Re: Kafka Exception

Hi, Kenan After studying the source code and searching google for related information, I think this should be caused by duplicate client_id [1], you can check if there are other jobs using the same group_id in consuming this topic. group_id is used in Flink to assemble client_id [2], if there are

Re: Suggestions for Open Source FLINK SQL editor

Hi Rajat, I also think Zeppelin may be a good choice. Best, Ron Guanghui Zhang 于2023年7月28日周五 20:45写道： > Hi, Guozhen, our team also use flink as ad-hoc query engine. Can we talk > aboat it > > Guozhen Yang 于2023年7月20日周四 11:58写道： > >> Hi Rajat, >> >> We are using apache zeppelin as our entry

Re: Set processing time in the past

Hi, Eugenio Can you describe the requirements in more detail? Best, Ron Shammon FY 于2023年7月17日周一 09:10写道： > Hi Eugenio, > > I cannot catch it clearly, could you describe it in more detail? > > Best, > Shammon FY > > On Sat, Jul 15, 2023 at 5:14 PM Eugenio Marotti < >

Re: how to get blackhole connector jar

hi longfeng， I think you should check the BlackHole connector related code in which module, then you can place this module jar to flink lib directory. Best, Ron Hang Ruan 于2023年6月28日周三 16:48写道： > Hi, longfeng, > > I check the blackhole connector document[1] and the blackhole connector is > a

Re: SQL-gateway Failed to Run

Hi, Xiaolong As Shammon says, I think you should the exception info of Flink cluster first to confirm the root cause. Best, Ron Shammon FY 于2023年7月4日周二 16:44写道： > Hi Xiaolong, > > I think you may need to check the error log in the flink cluster to find > out the root cause. > > Best, >

Re: Unsubscribe

Hi, As Hang say, please send an email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, Best, Ron Ragini Manjaiah 于2023年7月5日周三 13:36写道： > Unsubscribe > > On Tue, Jul 4, 2023 at 1:33 PM Bauddhik Anand wrote: > >> Unsubscribe >> >

Re: flink14 batch mode can read.iceberg table but stream mode can not

Hi, As Martijn says, you can find help in Iceberg community. Best, Ron Martijn Visser 于2023年6月5日周一 22:45写道： > Hi, > > This question is better suited for the Iceberg community, since they've > built the Flink-Iceberg integration. > > Best regards, > > Martijn > > On Wed, May 31, 2023 at 9:48 

Re: flink14 sql sink kafka error

Hi, You can give more exception info which can help to find the root cause. But according to the context you provided, as Hang says, it may be the Kafka Server problem. Best, Ron Hang Ruan 于2023年6月6日周二 10:46写道： > Hi, 湘晗刚, > > This error seem to be an error from the Kafka server. Maybe you

Re: In HA mode, support the same application run multiple jobs?

Hi, Your mean Application Mode or other modes? Best, Ron Weihua Hu 于2023年5月23日周二 21:29写道： > Hi > > High-Availability in Application Mode is only supported for > single-execute() applications.[1] > > And the reason is[2]: > > The added complexity stems mainly from the fact that the different

Re: Seeking advice on Looking up data - FileSystem connector / Kafka topic

Hi, Neha > 1. Is there a plan to extend filesystem connector to support lookup on the file in the way I mentioned above? Or if it already does, I would really appreciate if you could point me to some example or documentation. According to my understanding, there are no plans for community to

Re: Why I can't run more than 19 tasks?

Hi, Hemi, According to your context, it seems that the problem comes from MySQL client side, maybe you can check the client configuration. Best, Ron Shammon FY 于2023年5月25日周四 08:40写道： > Hi Hemi, > > There may be two reasons that I can think of > 1. The number of connections exceeds the MySQL

Re: [DISCUSS] Hive dialect shouldn't fall back to Flink's default dialect

Thanks for your proposal. I even don't notice this fallback behavior, +1. Best, Ron Jingsong Li 于2023年5月30日周二 15:23写道： > +1, the fallback looks weird now, it is outdated. > > But, it is good to provide an option. I don't know if there are some > users who depend on this fallback. > > Best, >

Re: Table API function and expression vs SQL

Hi, Flink SQL follows the standard SQL, so I think SQL syntax will be richer and richer in higher versions, it will not be changed. Best, Ron liu ron 于2023年4月13日周四 10:24写道： > Hi, > Flink SQL follows the standard SQL, so I think SQL syntax will be richer > and richer in higher

Re: Reading Parquet file with array of structs cause error

It will be addressed in FLINK-28867. Best, Ron Benenson, Michael via user 于2022年11月16日周三 08:47写道： > Thanks, Jing > > > > Do you know, if this problem will be addressed in FLINK-28867 > or it deserve a > separate Jira? > > > > > > *From:

Re: [blog article] Howto migrate a real-life batch pipeline from the DataSet API to the DataStream API

Thanks for your post, It looks very good to me, also maybe for developers, Best, Liudalong yuxia 于2022年11月8日周二 09:11写道： > Wow, cool! Thanks for your work. > It'll be definitely helpful for the users that want to migrate their batch > job from DataSet API to DataStream API. > > Best regards, >