Re: Problems with the state.backend.fs.memory-threshold parameter

Hi rui, The 'state.backend.fs.memory-threshold' configures the threshold below which state is stored as part of the metadata, rather than in separate files. So as a result the JM will use its memory to merge small checkpoint files and write them into one file. Currently the FLIP-306[1][2] is

Re: Cannot find metata file metadats in directory

After the task is restarted for several times, it is found that the supported cp is deleted. I view the audit log of HDFS and find that the deletion request comes from JM Hangxiang Yu 于2023年9月30日周六 17:10写道： > Hi, > How did you point out the checkpoint path you restored from ? > > Seems that you

Re: Cannot find metata file metadats in directory

After the task is restarted for several times, it is found that the supported cp is deleted. I view the audit log of HDFS and find that the deletion request comes from JM Hangxiang Yu 于2023年9月30日周六 17:10写道： > Hi, > How did you point out the checkpoint path you restored from ? > > Seems that you

Re: Kinesis Producer - Support separate Cloudwatch credentials

Hey, The FlinkKinesisProducer is deprecated in favour of the KinesisSink. The new sink does not rely on KPL, so this would not be a problem here. Is there a reason you are using the FlinkKinesisProducer instead of KinesisSink? Thanks for the deep dive, generally speaking I agree it would be

Re: Problems sinking a Flink table to Kafka

Hi, You are using a special Kafka connector. From the definition in website: "as a sink, the upsert-kafka connector can consume a changelog stream. It will write INSERT/UPDATE_AFTER data as normal Kafka messages value, and write DELETE data as Kafka messages with null values (indicate

Re: 关于Flink SQL无法对带metadata列的表在执行insert into时，指定具体的列名的问题。

感谢大佬！！！ > 2023年10月13日 10:44，tanjialiang 写道： > > Hi, > 这个问题已经在1.17.0修复，详细可以看https://issues.apache.org/jira/browse/FLINK-30922 > > > best wishes, > tanjialiang. > > > 回复的原邮件 > | 发件人 | jinzhuguang | > | 发送日期 | 2023年10月13日 10:39 | > | 收件人 | user-zh | > | 主题 | 关于Flink

Re: Flink SQL runtime

Hi, What SQL Runtime are you referring to? Why do you need to get it? Best, Ron Enric Ott <243816...@qq.com> 于2023年10月12日周四 14:26写道： > Hi,Team: > Is there any approach to get flink sql runtime via api ? > Any help would be appreciated. >

RE: Plans for upgrading Curl with latest 8.4.0

: Martijn Visser Sent: Thursday, October 12, 2023 3:24 PM To: Singhal, Ankur Cc: user@flink.apache.org Subject: Re: Plans for upgrading Curl with latest 8.4.0 Hi Ankur, Where do you see Flink using/bundling Curl? Best regards, Martijn On Wed, Oct 11, 2023 at 9:08 AM Singhal, Ankur wrote

Re: Plans for upgrading Curl with latest 8.4.0

Hi Ankur, Where do you see Flink using/bundling Curl? Best regards, Martijn On Wed, Oct 11, 2023 at 9:08 AM Singhal, Ankur wrote: > > Hi Team, > > > > Do we have any plans to update flink to support Curl 8.4.0 with earlier > versions having severe vulnerabilities? > > > > Thanks & Regards, >

Re: Flink with HDFS

Hi Jose, Sorry, my previous response may have been misleading. I have confirmed here that Flink 1.15 only supports Hadoop 2.8.5 and above, so you should use a Hadoop version of 2.8.5 or higher.

Re: Flink with HDFS

Thanks for the above reply. To the repesct, I comment the following: in tests performed with flink-shaded-hadoop-2-uber-2.8.3-10.0.jar, Hadoop 2.8.3 and Flink 1.10.0 everything goes fine. But, when I upgrade Flink to 1.15.0. I encounter errors. According to the documentation, Flink 1.15.0 only

Re: Running FlinkSQL locally

Hi Ralf, Have you tried _only_ adding flink-sql-connector-kafka-1.17.1.jar to the lib directory of your Flink deployment? It is a fat JAR, with all the dependencies shaded into it, so when you add them separately again, they will clash. Best Am Mi., 11. Okt. 2023 um 04:26 Uhr schrieb Ralph

Re: Metric to capture decoding failure in flink sources

Thanks a lot for your response Mason. Is there any FLIP planned to expose context in Reader in future? Regards, Prateek Kohli On Wed, Oct 11, 2023 at 6:03 AM Mason Chen wrote: > Hi Prateek, > > I agree, the reader should ideally expose the context to record metrics > about deserialization.

Re: Metric to capture decoding failure in flink sources

Hi Prateek, I agree, the reader should ideally expose the context to record metrics about deserialization. One option is to defer deserialization to another operator, say a RichMapFunction that has access to the RuntimeContext. Best, Mason On Mon, Oct 9, 2023 at 12:42 PM Prateek Kohli wrote:

RE: confirm subscribe to user@flink.apache.org

} } Any help would be appreciated. Thanks and regards,   ANITA KUMARI |  Software Engineer Sr  |  Insight t. 480-366-7025  |  c. 913.272.9568  |  anita.kum...@insight.com  |  insight.com  -Original Message- From: Kumari, Anita Sent: Monday, October 9, 2023 8:47 AM To: us

Re: observedGeneration field in FlinkDeployment

I think that a FLIP JIRA should be created to add an `observedGeneration` field to the spec. When I look at other kubernetes APIs, I see the `observedGeneration` field in many of them: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.26/ On Mon, Sep 11, 2023 at 11:51 AM Tony Chen

Re: flink两阶段提交

hi, 可以参考这篇博客，描述的非常清晰： https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/ Best, Feng On Sun, Sep 24, 2023 at 9:54 PM 海风 <18751805...@163.com> wrote: > 请教一下，flink的两阶段提交对于sink算子，预提交是在做检查点的哪个阶段触发的？预提交时具体是做了什么工作？ > > >

Re: Flink CDC消费Apache Paimon表

hi casel Flink 实时消费 paimon，默认情况就是全量 + 增量的方式。 具体可以参考： https://paimon.apache.org/docs/master/maintenance/configurations/ 中的 scan.mode 参数 best, Feng On Fri, Sep 29, 2023 at 5:50 PM casel.chen wrote: > 目前想要通过Flink全量+增量消费Apache Paimon表需要分别起离线和增量消费两个作业，比较麻烦，而且无法无缝衔接，能否通过类似Flink >

Re: How/where to check the operator id

Thank you Chen. There's no specific need so far for the info. I just wanted to check that the implementation (I mean assignment of operator IDs) has been done correctly and if I try to restore the job from a checkpoint with some changes in the job graph for other stateless operators, I would

Re: 退订

Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 来取消订阅邮件。 Best, Yunfeng On Wed, Oct 4, 2023 at 10:07 AM 1 wrote: > >

Re: Rolling back a bad deployment of FlinkDeployment on kubernetes

So, I was able to get the rollback to work after I changed my upgradeMode to *last-state*. Previously, my upgradeMode was *savepoint*, and when I deployed a bad commit, the jobmanager-leader configmap would get deleted. Once I changed the upgradeMode to *last-state*, the configmap was retained

Re: Continuous errors with Azure ABFSS

Yes, that also works correctly, at least based on the Kafka source we use (we'd get an alert if it suddenly started consuming from a very old offset). Regards, Alexis. On Thu, 5 Oct 2023, 19:36 ramkrishna vasudevan, wrote: > Sorry for the late reply. Just in case you restart the job , is it

Re: Using Flink k8s operator on OKD

Hey, We don’t have minimal supported version in the docs as we haven’t experienced any issue specific to kubernetes versions so far. We don’t really rely on any newer features Cheers Gyula On Fri, 6 Oct 2023 at 06:02, Krzysztof Chmielewski < krzysiek.chmielew...@gmail.com> wrote: > It seems

Re: Using Flink k8s operator on OKD

It seems that problem was caused by k8s 1.19. When we deployed Flink operator on vanilla k8s 1.19 we got the same error that we have on OKD 4.6.0 We are planing to update OKD to newer version that will use more up to date k8s. What is the minimal k8s version required for/supported by Flink

Re: Rolling back a bad deployment of FlinkDeployment on kubernetes

Hi Tony! There are still a few corner cases when the operator cannot upgrade / rollback deployments due to the loss of HA metadata (and with that checkpoint information). Most of these issues are not related to the operator logic directly but to how Flink handles certain failures and are related

Re: PyFlink MapState with Types.ROW() throws exception

After digging into the flink-python code, It seems if `PYFLINK_GATEWAY_DISABLED` is set to false in an environment variable, then using Types.LIST(Types.ROW([...])) does not have any issue, once Java Gateway is launched. It was unexpected for Flink local run to set this flag to false explicitly.

Re: Continuous errors with Azure ABFSS

Sorry for the late reply. Just in case you restart the job , is it able to safely use the checkpoint and get back to the checkpointed state? Regards Ram, On Thu, Sep 28, 2023 at 4:46 PM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hi Surendra, > > there are no exceptions in the

Re: Rolling back a bad deployment of FlinkDeployment on kubernetes

I tried this out with operator version 1.4 and it didn't work for me. I noticed that when I was deploying a bad version, the Kubernetes HA metadata and configmaps were deleted: [m [33m2023-10-05 14:52:17,493 [m [36mo.a.f.k.o.l.AuditUtils [m [32m[INFO ][flink-testing-service/flink-testing-service]

Re: Rolling back a bad deployment of FlinkDeployment on kubernetes

I just saw this experimental feature in the documentation: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#application-upgrade-rollbacks-experimental I'm guessing this is the only way to automate rollbacks for now. On Wed, Oct 4, 2023

Re: Pyflink unittest cases

. On Mon, Oct 2, 2023 at 9:21 PM joshua perez wrote: > Hello folks, > > Any help is appreciated. > > J. > > On Sat, Sep 30, 2023 at 1:47 PM joshua perez wrote: > >> Hi Team, >> >> We recently have started a use case where there would be involvement of >> Kafka and Flink's low level APIs like

Re: Custom Prometheus metrics disappeared in 1.16.2 => 1.17.1 upgrade

Hi Javier, Is there a particular reason why you aren't leveraging Flink metric API? It seems that functionality was internal to the PrometheusReporter implementation and your usecase should've continued working if it had depended on Flink's metric API. Best, Mason On Thu, Sep 28, 2023 at 2:51 

Re: Kafka Sink and Kafka transaction timeout

maximum_checkpoint_duration > But why adding maximum_restart_duration? > > If the application recovers from a checkpoint, any uncommitted message > that was written after the last successful checkpoint will be > re-written regardless. > If a transaction times out during the recov

Re: Watermarks

As per this link, it says that it only supports value_only for now as I am using pyflink. Does it mean I can't extract the timestamp appended by Kafka with pyflink as of now? https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/kafka/#deserializer or does it mean

Re: Unable to read records from kafka

Hi Team, you can ignore this thread. I was able to resolve this. J. On Mon, Oct 2, 2023 at 8:40 PM joshua perez wrote: > Hi team, > > I am trying to read the records from the Kafka topic and below is my very > basic code as of now > > from pyflink.datastream.connectors.kafka import

Re: Pyflink unittest cases

Hello folks, Any help is appreciated. J. On Sat, Sep 30, 2023 at 1:47 PM joshua perez wrote: > Hi Team, > > We recently have started a use case where there would be involvement of > Kafka and Flink's low level APIs like map and process functions and since I > am entirely new to these stuffs,

Re: Watermarks

Hi Liu and Jinfeng, I am trying to implement KafkaDeserializationSchema for Pyflink but am unable to get any examples. Can you share some links or references using which I can understand and try to implement myself? Perez sid.

Re: Flink Kafka offset commit issues

Hi Hangxiang, Thanks for providing me the steps to check if the checkpointing is getting triggered on failure recovery. I will follow them and respond back in case of any issues. Thanks, Elakiya On Sat, Sep 30, 2023 at 2:34 PM Hangxiang Yu wrote: > Hi, Elakiya. > > I think you could check : >

Re: Cannot find metata file metadats in directory

Hi, How did you point out the checkpoint path you restored from ? Seems that you are trying to restore from a not completed or failed checkpoint. On Thu, Sep 28, 2023 at 6:09 PM rui chen wrote: > When we use 1.13.2,we have the following error: > FileNotFoundException: Cannot find metata file

Re: Cannot find metata file metadats in directory

Hi, How did you point out the checkpoint path you restored from ? Seems that you are trying to restore from a not completed or failed checkpoint. On Thu, Sep 28, 2023 at 6:09 PM rui chen wrote: > When we use 1.13.2,we have the following error: > FileNotFoundException: Cannot find metata file

Re: Flink Kafka offset commit issues

Hi, Elakiya. I think you could check : 1. The TaskManager Log to figure out whether the job is restoring from an existing checkpoint and the restoring checkpoint path. 2. Or you could check the checkpoint ID when you restart your job (If not restoring from a checkpoint, it starts

Re: Flink Kafka offset commit issues

Hi Feng, Thanks for your response. 1. We have configured checkpointing to upload to a s3 location, also we see metadata files getting created in the s3 location. But we are unsure if the job is getting triggered from that checkpoint in case of failure. Is there a possible way to test this. Also

Re: Flink Kafka offset commit issues

Hi Elakiya 1. You can confirm if the checkpoint for the task has been triggered normally? 2. Also, If you stop the job, you need to use "STOP WITH SAVEPOINT" and specify the path to the savepoint when starting the Flink job for recovery. This is necessary to continue consuming from the

Re: Continuous errors with Azure ABFSS

Hi Surendra, there are no exceptions in the logs, nor anything salient with INFO/WARN/ERROR levels. The checkpoints are definitely completing, we even set the config execution.checkpointing.tolerable-failed-checkpoints: 1 Regards, Alexis. Am Do., 28. Sept. 2023 um 09:32 Uhr schrieb Surendra

Re: Continuous errors with Azure ABFSS

Hi Alexis, Could you please check the TaskManager log for any exceptions? Thanks Surendra On Thu, Sep 28, 2023 at 7:06 AM Alexis Sarda-Espinosa < sarda.espin...@gmail.com> wrote: > Hello, > > We are using ABFSS for RocksDB's backend as well as the storage dir > required for Kubernetes HA. In

Re: Custom Prometheus metrics disappeared in 1.16.2 => 1.17.1 upgrade

Thanks! I saw the first change but missed the third one, that is the most that most probably explains my problem, most probably the metrics I was sending with the twitter/finagle statsReceiver ended up in the singleton default registry and were exposed by Flink with all the other Flink metrics,

Re: Continuous errors with Azure ABFSS

Hi Ram, Thanks for that. We configure a path with ABFSS scheme in the following settings: - state.checkpoints.dir - state.savepoints.dir - high-availability.storageDir We use RocksDB with incremental checkpointing every minute. I found the metrics from Azure in the storage account under

Re: After using the jemalloc memory allocator for a period of time, checkpoint timeout occurs and tasks are stuck

hi Rui, We are using open jdk , `openjdk version "1.8.0_265"` Best, Feng On Thu, Sep 28, 2023 at 2:15 PM rui chen wrote: > hi Feng > > Are you using the open jdk or the oracle jdk? > > Best， > rui > > rui chen 于2023年9月27日周三 20:22写道： > >> hi Feng， >> >> Thanks for your reply, we are 8 u192

Re: After using the jemalloc memory allocator for a period of time, checkpoint timeout occurs and tasks are stuck

hi Feng Are you using the open jdk or the oracle jdk? Best， rui rui chen 于2023年9月27日周三 20:22写道： > hi Feng， > > Thanks for your reply, we are 8 u192 JDK, may be is the question, I found > a JDK issue:https://bugs.openjdk.org/browse/JDK-8215355. > > Best, > rui > > Feng Jin 于2023年9月27日周三

Re: Continuous errors with Azure ABFSS

Can you help with more info here? The RocksDB backend itself is in ABFS instead of local? Or you mean the checkpoint is in ABFS but local dir for RocksDB is in local storage? GetPathSTatus is done by your monitoring pages? We run Flink on ABFS so we would like to see if we can help you out.

Re: Flink with HDFS

Hi Jose This component is compatible with the latest version of Flink (1.17.1). The package in question is merely a shaded dependency for Hadoop and is not dependent on the version of Flink. Best, Feng On Thu, Sep 28, 2023 at 4:03 AM Jose Henrry Matituy Manchabajoy < henrrymati...@gmail.com>

Re: After using the jemalloc memory allocator for a period of time, checkpoint timeout occurs and tasks are stuck

hi Feng， Thanks for your reply, we are 8 u192 JDK, may be is the question, I found a JDK issue:https://bugs.openjdk.org/browse/JDK-8215355. Best, rui Feng Jin 于2023年9月27日周三 20:09写道： > hi Rui, > > Which version of JDK are you using? > > This issue could potentially be a bug in the JDK version.

Re: After using the jemalloc memory allocator for a period of time, checkpoint timeout occurs and tasks are stuck

hi Rui, Which version of JDK are you using? This issue could potentially be a bug in the JDK version. If you are using JDK 8, you can try using OpenJDK 8u265 as a possible solution. Best, Feng On Wed, Sep 27, 2023 at 8:08 PM rui chen wrote: > > > rui chen 于2023年9月27日周三 19:32写道： > >> hi

Re: After using the jemalloc memory allocator for a period of time, checkpoint timeout occurs and tasks are stuck

rui chen 于2023年9月27日周三 19:32写道： > hi Feng, > > Thank you for your reply，We observed the GC situation, there is no change > before and after replacement, several tasks on our line using jemalloc have > appeared stuck, after removing jemalloc, no stuck situation has been found. > > Best, > rui > >

Re: Custom Prometheus metrics disappeared in 1.16.2 => 1.17.1 upgrade

Have you checked the metric changes in 1.17. >From release notes 1.17: https://nightlies.apache.org/flink/flink-docs-master/release-notes/flink-1.17/ Metric Reporters # Only support reporter factories for instantiation # FLINK-24235 # Configuring reporters by their class is no longer supported.

Re: After using the jemalloc memory allocator for a period of time, checkpoint timeout occurs and tasks are stuck

hi rui, In general, checkpoint timeouts are typically associated with the job's processing performance. When using jemalloc, performance degradation is generally not observed. It is advisable to analyze whether the job's garbage collection (GC) has become more frequent. Best, Feng On Mon,

Re: Pending Metrics

Hi Rania Currently, there are no similar metrics available. Only the number of memory segments in the queue between tasks (inputQueueLength/outputQueueLength) and the buffer occupancy ratio (inPoolUsage/outPoolUsage) are available. For more detailed information, please refer to:

Re: Custom Prometheus metrics disappeared in 1.16.2 => 1.17.1 upgrade

Some more details on my problem: 1. The "Multiple implementations" problem was because I had the metrics-prometheus jar both in the plugins and lib directories. I tried putting it in only one, and in both cases (plugins or lib), the result was the same, I got only Flink metrics on my prom port.

Re: Securing Keytab File in Flink

Thanks Gabor, reducing the attack vector looks a fair call here. However, I am still thinking of other ways to eliminate this security concern. Is there a way I can use ticketCache inside my pods somehow? Maybe something like Yarn?  Just thinking out loud, but would there be a case of

Re: Is there any example that use hbase connector in stream API

HI Антон,Hang Thanks Reply I implemented the RichSinkFunction with HbaseClient ,it works well. I plan to copy some code from org.apache.flink.connector.hbase.sink.HBaseSinkFunction mainly the scheduled batch submit ,but except the converter part . I think I can read from TableAPI and write with

Re: Is there any example that use hbase connector in stream API

Hi, I wrote to hbase using map extending RichMapFunction. I initialized the connection in overrided open method and wrote to hbase in overrided map method. 22.09.2023, 05:01, "碳酸君" :hi community: I'm trying to write some data to hbase in a stream job ,withflink-connector-hbase-2.2 . I have

Re: 退订

Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org mailto:u...@flink.apache.org>> 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org

Re: 退订

Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org mailto:u...@flink.apache.org>> 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org

Re: 退订

Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org mailto:u...@flink.apache.org>> 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org

Re: Side outputs documentation

I see, sounds good, thanks for the clarification. Am Di., 26. Sept. 2023 um 03:29 Uhr schrieb Yunfeng Zhou < flink.zhouyunf...@gmail.com>: > Hi Alexis, > > Thanks for the clarification. I found the second constructor on > Flink's master branch here[1], and maybe it was that we had been >

Re: Is there any example that use hbase connector in stream API

Hi, There is not a HBase connector for the DataStream API. But we could follow codes in the Table API to learn how to use it. 1. We could know how to build a HBaseTableSchema in HBase2DynamicTableFactory[1]. 2. We could know how to build a RowDataToMutationConverter in HBaseDynamicTableSink[2].

Re: Side outputs documentation

Hi Alexis, Thanks for the clarification. I found the second constructor on Flink's master branch here[1], and maybe it was that we had been commenting on different versions of Flink, and the second constructor has not been introduced in the version you use. From the source code I can see that the

Re: Default Flink S3 FileSource timeout due to large file listing

, applies transformations and re-writes them to S3, under a different location. Below is my code to read from parquets from S3: ``` final Configuration configuration = new Configuration(); configuration.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSys

Re: Flink upgrade 1.14.6 to 1.15.4 -> StackOverflowError

: Thursday, September 14, 2023 6:42:29 PM To: Flink User Mail List Subject: Re: Flink upgrade 1.14.6 to 1.15.4 -> StackOverflowError Hi, Alexey Have you tried other versions like 1.15.3 or 1.16.1? Best, Ron Alexey Trenikhun mailto:yen...@msn.com>> 于2023年9月15日周五 04:24写道： Hello, After upgrad

Re: Side outputs documentation

Hi Yunfeng, Thanks for the response. I hadn't even seen the other constructor, but it seems that the single-arg constructor works fine even if the output tag is declared as "static final", at least in my use case. I imagine Flink would complain about unknown types if it really can't figure it out

Re: Flink File Source: File read strategy

hat File Source reads files directly from S3? > > > > Regards, > > Kirti Dhar > > > > *From:* Shammon FY > *Sent:* 25 September 2023 06:27 > *To:* Kirti Dhar Upadhyay K > *Cc:* user@flink.apache.org > *Subject:* Re: Flink File Source: File read strategy &

Re: 1.17.1 - Interval join的时候发生NPE

Hi Hangxiang, 感谢您的回应。 下面是该问题的关键代码，main_stream表是流数据源，数据事件流频约每笔500ms~1s，目前尝试将t1minStream和t5minStream assignTimestampsAndWatermarks(WatermarkStrategy.noWatermarks())是不会产生这问题造成作业失败了，但输出会有数据丢失。 如有其他思路，麻烦你了。 String t1minSql = "SELECT rowtime, key, id, AVG(num) OVER w_t1min AS avg_t1min FROM

RE: Flink File Source: File read strategy

Thanks Shammon. Is there any way to verify that File Source reads files directly from S3? Regards, Kirti Dhar From: Shammon FY Sent: 25 September 2023 06:27 To: Kirti Dhar Upadhyay K Cc: user@flink.apache.org Subject: Re: Flink File Source: File read strategy Hi Kirti, I think the default

Re: Flink SQL的状态清理

Hi, 可以通过设置 table.exec.state.ttl 来控制状态算子的 state TTL. 更多信息请参阅 [1] [1] https://nightlies.apache.org/flink/flink-docs-master/zh/docs/dev/table/concepts/overview/#%e7%8a%b6%e6%80%81%e7%ae%a1%e7%90%86 Best, Jane On Thu, Sep 21, 2023 at 5:17 PM faronzz wrote: > 试试这个

Re: 退订

Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user-zh@flink.apache.org , and you can refer [1][2] for more details. Best,

Re: Side outputs documentation

Hi Alexis, If you create OutputTag with the constructor `OutputTag(String id)`, you need to make it anonymous for Flink to analyze the type information. But if you use the constructor `OutputTag(String id, TypeInformation typeInfo)`, you need not make it anonymous as you have provided the type

Re: 1.17.1 - Interval join的时候发生NPE

Hi, 请问下是 SQL 作业还是 DataStream 作业，可以提供一些可复现的关键 SQL 或代码吗 On Sat, Sep 23, 2023 at 3:59 PM Phoes Huang wrote: > Hi, > > 单机本地开发执行，遇到该问题，请问有人遇过并解决吗？ > > 2023-09-23 13:52:03.989 INFO > [flink-akka.actor.default-dispatcher-9][Execution.java:1445] - Interval > Join (19/20) >

Re: About Flink parquet format

ly without closing? > > > > Rgds, > > Kamal > > > > *From:* Kamal Mittal via user > *Sent:* 21 September 2023 07:58 AM > *To:* Feng Jin > *Cc:* user@flink.apache.org > *Subject:* RE: About Flink parquet format > > > > Yes. > > > > Du

Re: Re: Re: How to read flinkSQL job state

Hi, Yifan. Unfortunately, IIUC, we could get the key and value type only by reading related sql codes currently. I think it's useful if we could support SQL semantics for the Processor API, but it indeed will take lots of effort. On Thu, Sep 21, 2023 at 12:05 PM Yifan He via user wrote: > Hi

RE: About Flink parquet format

Hello, Can you please share that why Flink is not able to handle exception and keeps on creating files continuously without closing? Rgds, Kamal From: Kamal Mittal via user Sent: 21 September 2023 07:58 AM To: Feng Jin Cc: user@flink.apache.org Subject: RE: About Flink parquet format Yes

Re: Flink File Source: File read strategy

Hi Kirti, I think the default file `Source` does not download files locally in Flink, but reads them directly from S3. However, Flink also supports configuring temporary directories through `io.tmp.dirs`. If it is a user-defined source, it can be obtained from FlinkS3FileSystem. After the Flink

Re: Issue with flink-kubernetes-operator not updating execution.savepoint.path after savepoint deletion

Hi Operator savepoint retention and savepoint upgrades have nothing to do with each other I think. Retention is only for periodic savepoints triggered by the operator itself. I would upgrade to the latest 1.6.0 operator version before investigating further. Cheers Gyula On Sat, 23 Sep 2023 at

Re: Issue with flink-kubernetes-operator not updating execution.savepoint.path after savepoint deletion

Small update on this. I see that the issue is that we use `upgradeMode: savepoint`, but have not configured the operator to retain savepoints for long enough (the previous operator we used never deleted savepoints so we didn't run into this). I am reconfiguring to use `upgradeMode: last-state` and

Re: Window aggregation on two joined table

ain, not sure how to configure this > > Ancora cari saluti > > Thias > > > > > > From: Eugenio Marotti > Sent: Thursday, September 21, 2023 2:35 PM > To: Schwalbe Matthias > Cc: user@flink.apache.org > Subject: Re: Window aggregation on tw

Re: Zookeeper HA with Kubernetes: Possible to use the same Zookeeper cluster w/multiple Flink Operators?

Gyula, Thanks, you've helped us move closer to migrating our application to the Flink Operator![0] Best, Brian King SRE, Data Platform/Search Platform Wikimedia Foundation IRC: inflatador [0] https://phabricator.wikimedia.org/T326409 > On Sep 21, 2023, at 12:16 AM, Gyula Fóra wrote: > >

RE: Window aggregation on two joined table

in which case it is rendered as ‘timeless’ and does not prevent time progress in your join operator * Again, not sure how to configure this Ancora cari saluti Thias From: Eugenio Marotti Sent: Thursday, September 21, 2023 2:35 PM To: Schwalbe Matthias Cc: user@flink.apache.org Subject: Re

Re: Window aggregation on two joined table

Hi Matthias, No the second table doesn’t have an event time and a watermark specified. In order for the window to work do I need a watermark also on the second table? Thanks Eugenio > Il giorno 21 set 2023, alle ore 13:45, Schwalbe Matthias > ha scritto: > > Ciao Eugenio, > > I might be

RE: Window aggregation on two joined table

Ciao Eugenio, I might be mistaken, but did you specify the event time for the second table like you did for the first table (watermark(….))? I am no so acquainted with table api (doing more straight data stream api work), but I assume this join and windowing should be by event time. What do

Re: Zookeeper HA with Kubernetes: Possible to use the same Zookeeper cluster w/multiple Flink Operators?

Hi! The cluster-id for each FlinkDeployment is simply the name of the deployment. So they are all different in a given namespace. (In other words they are not fixed as your question suggests but set automatically) So there should be no problem sharing the ZK cluster . Cheers Gyula On Thu, 21

RE: Re: Re: How to read flinkSQL job state

Hi Hangxiang, I still have one question about this problem, when using datastream api I know the key and value type I use in state because I defined ValueStateDescriptor, but how can I get the ValueStateDescriptor in flinksql? Thanks, Yifan On 2023/09/07 06:16:41 Hangxiang Yu wrote: > Hi,

RE: About Flink parquet format

Yes. Due to below error, Flink bulk writer never close the part file and keep on creating new part file continuously. Is flink not handling exceptions like below? From: Feng Jin Sent: 20 September 2023 05:54 PM To: Kamal Mittal Cc: user@flink.apache.org Subject: Re: About Flink parquet

Re: Flink cdc 2.0 历史数据太大，导致log积压怎么解决

你好，除了这些运维手段外，flink cdc本身有什么解法吗，比如说增量阶段不用从头开始读binlog，因为其实很多都是重复读到的数据 > 2023年9月20日 21:00，Jiabao Sun 写道： > > Hi, > 生产环境的binlog还是建议至少保留7天，可以提高故障恢复时间容忍度。 > 另外，可以尝试增加snapshot的并行度和资源来提升snapshot速度，snapshot完成后可以从savepoint恢复并减少资源。 > Best, > Jiabao >

Re: 回复：flink1.17版本不支持hive 2.1版本了吗

把这个 pr https://github.com/apache/flink/pull/19352 revert 掉，然后重新打包 flink hive connector 就可以。 Best regards, Yuxia - 原始邮件 - 发件人: "迎风浪子" <576637...@qq.com.INVALID> 收件人: "user-zh" 发送时间: 星期二, 2023年 9 月 19日 下午 5:20:58 主题: 回复：flink1.17版本不支持hive 2.1版本了吗 我们还在使用hive1.1.0，怎么办？ ---原始邮件--- 发件人:

Re: Using Flink k8s operator on OKD

Thank you Zach, our flink-operator and flink deployments are in same namespace -> called "flink". We have executed what is described in [1] before my initial message. We are using OKD 4.6.0 that according to the doc is using k8s 1.19. the very same config is working fine on "vanilla" k8s, but for

Re: About Flink parquet format

> This is coming from class CapacityByteArrayOutputStream and contained in > parquet-common library. > > > > Rgds, > > Kamal > > > > *From:* Feng Jin > *Sent:* 19 September 2023 01:01 PM > *To:* Kamal Mittal > *Cc:* user@flink.apache.org > *Subject:* Re: About Fli

Re: Flink cdc 2.0 历史数据太大，导致log积压怎么解决

Hi, 生产环境的binlog还是建议至少保留7天，可以提高故障恢复时间容忍度。 另外，可以尝试增加snapshot的并行度和资源来提升snapshot速度，snapshot完成后可以从savepoint恢复并减少资源。 Best, Jiabao -- From:jinzhuguang Send Time:2023年9月20日(星期三) 20:56 To:user-zh Subject:Flink cdc 2.0 历史数据太大，导致log积压怎么解决

Re: Using Flink k8s operator on OKD

I haven’t used OKD but it sounds like OLM. If that’s the case, I’m assuming the operator was deployed to the “operators” namespace. In that case, you’ll need to create the RBACs and such in the Flink namespace for that deployment to work. For example this needs to be in each namespace that you

Re: Incompatible KafkaProducer version

Hi Krzysztof, The bundled Flink Kafka connector for 1.17 uses Kafka 3.2.3, see https://github.com/apache/flink/blob/release-1.17/flink-connectors/flink-connector-kafka/pom.xml#L38 That's also the case for the externalized Flink Kafka connector v3.0, see

Re: [ANNOUNCE] Apache Flink Stateful Functions Release 3.3.0 released

Thanks Martijn! On Tue, Sep 19, 2023 at 6:38 AM Martijn Visser wrote: > The Apache Flink community is excited to announce the release of > Stateful Functions 3.3.0! > > Stateful Functions is a cross-platform stack for building Stateful > Serverless applications, making it radically simpler to

Re: flink design pazzule

Hi Zhuliang, I would suggest u reading the comments from 'ExternallyInducedSource.java'[1]. "Sources that implement this interface do not trigger checkpoints when receiving a trigger message from the checkpoint coordinator, but when their input data/events indicate that a checkpoint should be

RE: About Flink parquet format

To: Kamal Mittal Cc: user@flink.apache.org Subject: Re: About Flink parquet format Hi Kamal What exception did you encounter? I have tested it locally and it works fine. Best, Feng On Mon, Sep 18, 2023 at 11:04 AM Kamal Mittal mailto:kamal.mit...@ericsson.com>> wrote: Hello, Checkpo

<    6   7   8   9   10   11   12   13   14   15   >