Re: 退订

请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-lists 爱看书不识字 于2024年5月11日周六 10:06写道：

Re: Incremental snapshot based source

Hi. cloud young. The property 'log.mining.strategy' is actually a setting from the oracle debezium connector[1]. Are some exceptions able to be found in your job and which version are you using? Best, Hang [1]

Re: Elasticsearch8 example

Hi Tauseef. I see that the support of Elasticsearch 8[1] will be released in elasticsearch-3.1.0. So there is no docs for the elasticsearch8 by now. We could learn to use it by some tests[2] before the docs is ready. Best, Hang [1] https://issues.apache.org/jira/browse/FLINK-26088 [2]

Re: Table Source from Parquet Bug

Hi, David. Have you added the parquet format[1] dependency in your dependencies? It seems that the class ParquetColumnarRowInputFormat cannot be found. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/table/formats/parquet/ Sohil Shah 于2024年4月17日周三

Re: How to handle tuple keys with null values

Hi Sachin. I think maybe we could cast the Long as String to handle the null value. Or as Asimansu said, try to filter out the null data. Best, Hang Asimansu Bera 于2024年4月3日周三 08:35写道： > Hello Sachin, > > The same issue had been reported in the past and JIRA was closed without > resolution. >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

- GitHub Repository: https://github.com/apache/flink-cdc >>>> - Flink CDC Documentation: >>>> https://nightlies.apache.org/flink/flink-cdc-docs-stable >>>> >>>> After Flink community accepted this donation[1], we have completed >>>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

- GitHub Repository: https://github.com/apache/flink-cdc >>>> - Flink CDC Documentation: >>>> https://nightlies.apache.org/flink/flink-cdc-docs-stable >>>> >>>> After Flink community accepted this donation[1], we have completed >>>

Re: can we use Scan Newly Added Tables without restarting the existing job ?

Hi, 3pang zhu. This `Scan Newly added tables` feature requires restarting the job from the savepoint. We cannot add new tables to the running job without restarting by now. Best, Hang 3pang zhu 于2024年3月20日周三 15:22写道： > this link has describe the usage for [Scan Newly Added Tables] >

Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

Hi, Sachin. I use the command `jar -tf flink-dist-1.18.0.jar| grep OutputTag` to make sure that this class is packaged correctly. I think you should check your own jar to make sure this class is not packaged in your jar. Best, Hang Sachin Mittal 于2024年3月12日周二 20:29写道： > I miss wrote. It’s

Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR

Hi, Sachin. This error occurs when there is class conflict. There is no need to package flink-core in your own jar. It is already contained in flink-dist. And Flink version 1.8 is too old. It is better to update your flink version. Best, Hang Sachin Mittal 于2024年3月12日周二 16:04写道： > Hi, > We

Re: 退订

请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh-unsubscr...@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-lists willluzheng

Re: 退订

请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-lists 熊柱 <18428358...@163.com>

Re: 退订

请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh-unsubscr...@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-lists 王新隆 于2024年3月11日周一

Re: Entering a busy loop when adding a new sink to the graph + checkpoint enabled

iliar with kafka connector, I can't give you > an accurate reply. I think Hang Ruan is an expert in this field :). > > Hi, Ruan Hang. Can you take a look at this strange situation？ > > > -- > Best！ > Xuyang > > > 在 2024-03-10 16:49:16，"Daniel Peled" 写

Re: SecurityManager in Flink

Hi, Kirti. Could you please provide the stack trace of this NPE? I check the code and I think maybe the problem lies in LocalFileSystem#listStatus. The code in line 161[1] may return null, which will let LocalFileSystem#listStatus return null. Then the `containedFiles` is null and the NPE occurs.

Re: 退订

请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh-unsubscr...@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-lists 4kings...@gmail.com

Re: flink cdc底层的debezium是如何注册schema到confluent schema registry的？

Hi，casel.chen。 这个部分应该是在 CDC 项目里没有涉及到，CDC 依赖 debezium 的 engine 部分直接读取出变更数据，并没有像 debezium 本身一样去写入到 Kafka 中。 可以考虑去 Debezium 社区咨询一下这部分的内容，Debezium开发者们应该更熟悉这部分的内容。 祝好， Hang casel.chen 于2024年2月29日周四 18:11写道： > 搜索了debezium源码但没有发现哪里有调用 > SchemaRegistryClient.register方法的地方，请问它是如何注册schema到confluent

Re: mysql cdc streamapi与sqlapi 输出表现不相同

你好，ha.fengqi。 MySQL CDC 连接器只有在多并发时，会依赖checkpoint的完成来切换到增量阶段。从你提供的代码上来看，是单并发的运行作业，所以应该Source 在这两者之间的行为不会有区别。 这个不同是不是有可能是下游在两种使用方式上，有什么区别？ 可以通过观察具体的IO指标看到Source是否真的及时发出消息，如果比较熟悉代码，也可以自己添加一下打印日志来验证。 祝好， Hang

Re: DESCRIBE CATALOG not available?

Hi, Robin. I see that the `DESCRIBE CATALOG` sql is not list in the DESCRIBE document[1]. It is not available. Besides this, I checked the changes in Catalog.java from commits on May 9, 2019. I cannot find the method `explainCatalog` introduced from this FLIP. This FLIP is not finished yet.

Re: flink cdc 读取数据后面可以跟窗口函数吗

你好， CDC Source 目前不支持窗口函数。 不过可以考虑通过非窗口聚合的方式实现类似的效果。具体方法为： 1. 使用DATE_FORMAT函数，将时间字段转换成分钟粒度的字符串，作为窗口值。 2. 根据窗口值进行GROUP BY聚合。 Best, Hang Xuyang 于2024年1月17日周三 19:34写道： > Hi, > Flink SQL中可以用Group Window[1]的方式来读完cdc数据后加窗口。 > 可以具体描述一下“一直不生效”的现象和SQL么？ > > > > [1] >

Re: 退订

请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh-unsubscr...@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Jiabao [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-lists yd c

Re: flink-connector-dynamodb 4.2.0-1.18 does not provide flink-connector-base dependency

Hi, Tamir. This is an expected behavior. The flink-connector-base is already included in flink-dist and we will not package it in the externalized connectors. You could see this issue[1] for more details. Best, Hang [1] https://issues.apache.org/jira/browse/FLINK-30400?filter=-1 Tamir Sagi

Re: Re: Flink CDC MySqlSplitReader问题

t; > > > > > > > > > 在 2023-12-20 21:40:05，"Hang Ruan" 写道： > >Hi，casel > > > >这段逻辑应该只有在处理到新增表的时候才会用到。 > >CDC 读取数据正常是会在所有SnapshotSplit读取完成后，才会下发BinlogSplit。 > >但是如果是在增量阶段重启作业，同时新增加了一些表，就会出现同时有BinlogSplit和SnapshotSplit的情况，此时才会走到这段逻辑。 &

Re: Flink CDC MySqlSplitReader问题

Hi，casel 这段逻辑应该只有在处理到新增表的时候才会用到。 CDC 读取数据正常是会在所有SnapshotSplit读取完成后，才会下发BinlogSplit。 但是如果是在增量阶段重启作业，同时新增加了一些表，就会出现同时有BinlogSplit和SnapshotSplit的情况，此时才会走到这段逻辑。 Best， Hang key lou 于2023年12月20日周三 16:24写道： > 意思是当 有 binlog 就意味着 已经读完了 snapshot > > casel.chen 于2023年12月19日周二 16:45写道： > > >

Re: 退订

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from u...@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 u...@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: 退订

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: Flink KafkaProducer Failed Transaction Stalling the whole flow

Hi, Dominik. >From the code, your sink has received an InvalidTxnStateException in KafkaCommittter[1]. And kafka connector treats it as a known exception to invoke `signalFailedWithKnownReason`. `signalFailedWithKnownReason` will not throw an exception. It let the committer to decide fail or

Re: Is the kafka-connector doc missing a dependency on flink-connector-base

Hi, Jean-Marc Paulin. The flink-connector-base will not be packaged in the externalized connectors [1]. The flink-connector-base has been included in flink-dist and we should use the provided scope in maven for it. Best, Hang [1] https://issues.apache.org/jira/browse/FLINK-30400 Jean-Marc

Re: Elasticsearch connector support?

Hi, Lasse. There is already a discussion about the connector releases for 1.18[1]. Best, Hang [1] https://lists.apache.org/thread/r31f988m57rtjy4s75030pzwrlqybpq2 Lasse Nedergaard 于2023年11月24日周五 22:57写道： > Hi > > From the documentation I can see there isn’t any ES support in Flink 1.18 >

Re: Confluent Kafka conection error

Hi, Tauseef. This error is not that you can not access the Kafka cluster. Actually, this error means that the JM cannot access its TM. Have you ever checked whether the JM is able to access the TM? Best, Hang Tauseef Janvekar 于2023年11月23日周四 16:04写道： > Dear Team, > > We are facing the below

Re: The generated schema is not correct when using filesystem connector and avro format

Hi, julia. I have read the code about this part. The problem as you said is that the RowType passed to the avro-confluent format is nullable, which will cause union with null in the schema. I think FLINK-30438 is the same problem as yours. But I find the RowType passed to avro-confluent format in

Re: Handling default fields in Avro messages using Flink SQL

Hi, Dale. I think there are two choices to try. 1. As the reply in #22427[1], use the SQL function `COALESCE`. 2. Modify the code in Avro format by yourself. There is some work to do for the choice 2. First, you need to pass the default value in Schema, which does not contain the default value

Re: Kafka Topic Permissions Failure

Hi, Razin. It seems like the issue you shared is a different problem from yours. They have different error messages. Have you ever tried to consume this topic using the Kafka java client[1] by yourself to make sure you could access the topic normally? Best, Hang [1]

Re: Flink Job Failed With Kafka Exception

Hi, Madan. This error seems like that there are some problems when the consumer tries to read the topic metadata. If you use the same source for these topics, the kafka connector cannot skip one of them. As you say, you need to modify the connector's default behavior. Maybe you should read the

Re: [DISCUSS][FLINK-33240] Document deprecated options as well

Thanks for the proposal. +1 from my side and +1 for putting them to a separate section. Best, Hang Samrat Deb 于2023年11月1日周三 15:32写道： > Thanks for the proposal , > +1 for adding deprecated identifier > > [Thought] Can we have seperate section / page for deprecated configs ? Wdut > ? > > >

Re: [ANNOUNCE] Apache Flink 1.18.0 released

Congratulations! Best, Hang Samrat Deb 于2023年10月27日周五 11:50写道： > Congratulations on the great release > > Bests, > Samrat > > On Fri, 27 Oct 2023 at 7:59 AM, Yangze Guo wrote: > > > Great work! Congratulations to everyone involved! > > > > Best, > > Yangze Guo > > > > On Fri, Oct 27, 2023 at

Re: [ANNOUNCE] Apache Flink 1.18.0 released

Congratulations! Best, Hang Samrat Deb 于2023年10月27日周五 11:50写道： > Congratulations on the great release > > Bests, > Samrat > > On Fri, 27 Oct 2023 at 7:59 AM, Yangze Guo wrote: > > > Great work! Congratulations to everyone involved! > > > > Best, > > Yangze Guo > > > > On Fri, Oct 27, 2023 at

Re: Unsubscribe from user list.

Hi, Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. Best, Hang [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2]

Re: Is there any example that use hbase connector in stream API

Hi, There is not a HBase connector for the DataStream API. But we could follow codes in the Table API to learn how to use it. 1. We could know how to build a HBaseTableSchema in HBase2DynamicTableFactory[1]. 2. We could know how to build a RowDataToMutationConverter in HBaseDynamicTableSink[2].

Re: 退订

Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user-zh@flink.apache.org , and you can refer [1][2] for more details. Best,

Re: 退订

Hi, 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Please send email to user-zh-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user-zh@flink.apache.org , and you can refer [1][2] for more details. Best,

Re: Incompatible KafkaProducer version

Hi, Krzysztof. I find that this part has been changed in PR[1] when updating the kafka client version to 3.4.0. This fix is not released yet. Maybe you can package and check it by yourself. Best, Hang [1] https://github.com/apache/flink-connector-kafka/pull/11 Krzysztof Jankiewicz

Re: How to read flinkSQL job state

Hi, Yifan. I think the document[1] means to let us convert the DataStream to the Table[2]. Then we could handle the state with the Table API & SQL. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/libs/state_processor_api/ [2]

Re: How to trigger process function when no event in eventTimeWindow ?

Hi, longfent. We could use `withIdleness`[1] to deal with the idle sources. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources longfeng Xu 于2023年8月27日周日 14:01写道： > Hello, > > The issue I’m

Re: How to use pipeline.jobvertex-parallelism-overrides property.

Hi, Krzysztof. As liu ron said, the key of the map for this configuration is the value from JobVertexId#toHexString. Maybe we could improve the docs to provide more details. The condition that two operators have the same parallelism is a must for chaining them. If they have different

Re: Re-start strategy without checkpointing enabled

Hi, Kamal. If we don't enable checkpointing, the job will be started with the startup mode each time. For example, the job reads Kafka from the earliest offset and writes to mysql. If the job failover without checkpointing, the tasks will consume Kafka from the earliest offset again. I think it

Re: [Question] Good way to monitor data skewness

Hi, Dennis. As Ron said, we could judge this situation by the metrics. We are usually reporting the metrics to the external system like Prometheus by the metric reporter[1]. And these metrics could be shown by some other tools like grafana[2]. Best, Hang [1]

Re: Conversion expects insert-only records but DataStream API record contains: UPDATE_BEFORE

Hi, Changelog mode is the concept of the table API. You can get a changelog stream from StreamTableEnvironment#fromChangelogStream. For kafka connector, its changelog mode depends on the used format. Best, Hang liu ron 于2023年8月13日周日 22:06写道： > Hi, > > After deep dive into the source code, I

Re: Global/Shared objects

Hi, Kamal. Each TaskManager is a JVM process and each task slot is a thread of the TaskManager. More information see [1]. The static fields could be shared among subtasks in the same TaskManager. If the subtasks are running in the different TaskManager, they cannot share the static fields. Best,

Re: In Flink, is there a way to merge two streams in stateful manner

ps: Forget the link: Hybrid Source[1] [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/hybridsource/ Hang Ruan 于2023年8月11日周五 10:14写道： > Hi, Muazim. > > I think the Hybird Source[1] may be helpful for your case. > > Best, > Hang > &g

Re: In Flink, is there a way to merge two streams in stateful manner

Hi, Muazim. I think the Hybird Source[1] may be helpful for your case. Best, Hang Ken Krugler 于2023年8月11日周五 04:18写道： > As (almost) always, the devil is in the details. > > You haven’t said, but I’m assuming you’re writing out multiple files, each > with a different schema, as otherwise you

Re: Custom connector Update flink 1.13.3 failed notsuchMethod

Hi, longfeng. I think you should rebuild your connector according to the new API. The return type of the method `DynamicTableFactory$Context.getCatalogTable()` is already changed from `CatalogTable` to `ResolvedCatalogTable`[]. Best, Hang [1] https://issues.apache.org/jira/browse/FLINK-21913

Re: Kafka Exception

Hi, Kenan. Maybe you should set the `client.id.prefix` to avoid the conflict. Best, Hang liu ron 于2023年7月31日周一 19:36写道： > Hi, Kenan > > After studying the source code and searching google for related > information, I think this should be caused by duplicate client_id [1], you > can check if

Re: Flink netty connector for TCP source

Hi, Kamal. The SplitEnumerator is contained in the SourceCoordinator. They are only used in JM. Best, Hang Kamal Mittal via user 于2023年8月1日周二 10:43写道： > Thanks. > > > > I looked at the link for custom data sources, one query here that how to > make sure for Split enumerator to execute on Job

Re: Logs of Kafka payload in Table Api connector of Apache Flink

Hi. elaloya. If you want to log some information about the kafka records, you can add some logs in KafkaRecordEmitter. If you want to know the information about the deserialized value, you should add logs in the avro format. Best, Hang elakiya udhayanan 于2023年7月19日周三 19:44写道： > Hi Team, > > I

Re: Query on Flink SQL primary key for nested field

elakiya udhayanan 于2023年7月10日周一 16:41写道： > Hi Hang, > > The select query works fine absolutely, we have also implemented join > queries which also works without any issues. > > Thanks, > Elakiya > > On Mon, Jul 10, 2023 at 2:03 PM Hang Ruan wrote: > >> Hi,

Re: Query on Flink SQL primary key for nested field

Hi, Elakiya. Maybe this DDL could be executed. Please execute the select sql `select * from KafkaTable`. Then I think there will be some error or the `user_id` will not be read correctly. Best, Hang elakiya udhayanan 于2023年7月10日周一 16:25写道： > Hi Hang Ruan, > > Thanks for your

Re: Query on Flink SQL primary key for nested field

Hi, elakiya. The upsert-kafka connector will read the primary keys from the Kafka message keys. We cannot define the fields in the Kafka message values as the primary key. Best, Hang elakiya udhayanan 于2023年7月10日周一 13:49写道： > Hi team, > > I have a Kafka topic named employee which uses

Re: Recursive Split Detection + same split optimization

Hi, Benoit. A split enumerator responsible for discovering the source splits, and assigning them to the reader. It seems like that your connector discovering splits in TM and assigning them in JM. I think there are 2 choices: 1. If you need the enumerator to assign splits, you have to send the

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

Hi, Leonard. I would like to help to add this page. Please assign this issue to me. Thanks. Best, Hang Leonard Xu 于2023年7月7日周五 11:26写道： > Congrats to all ! > > It will be helpful to promote Apache Flink if we can add a page to our > website like others[2]. I’ve created an issue to improve

Re: Difference between different values for starting offset

Hi, Oscar. Yes, your are right. If starting from a checkpoint or savepoint, kafka connector will always use the offset stored in the states. If starting without a checkpoint or savepoint, kafka connector will use the specific startup mode. Best, Hang Oscar Perez via user 于2023年7月4日周二 20:54写道：

Re: Part files generated in reactive mode

Hi, Wang Mengxi, You should use the file compaction feature[1] to compact the small files. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/#compaction Wang, Mengxi X via user 于2023年7月4日周二 22:44写道： > Hi, > > > > We want to

Re: Unsubscribe

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: Unsubscribe

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: Unsubscribe

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: Unsubscribe

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: Unsubscribe

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: how to get blackhole connector jar

Hi, longfeng, I check the blackhole connector document[1] and the blackhole connector is a build-in connector. If you can not find this connector in your flink, maybe you should check the `flink-table-api-java-bridge` jar to find the `BlackHoleTableSinkFactory`[2]. Best, Hang [1]

Re: Unsubscribe

Please send email to user-unsubscr...@flink.apache.org if you want to unsubscribe the mail from user@flink.apache.org, and you can refer [1][2] for more details. 请发送任意内容的邮件到 user-unsubscr...@flink.apache.org 地址来取消订阅来自 user@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Hang [1]

Re: Admin Client Configs

Hi, Razin. You could pass these Kafka configuration in the format 'properties.*'. For example, 'properties.allow.auto.create.topics' = 'false'. See more in Kafka connector docs[1]. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/kafka/#properties

Re: Unsubscribe

Please send an email to user-unsubscr...@flink.apache.org to unsubscribe Best, Hang Yu voidy 于2023年6月12日周一 11:39写道： > >

Re: Custom Counter on Flink File Source

Kirti Dhar Upadhyay K 于2023年6月7日周三 15:49写道： > Thanks Hang. > > Any expected date for Flink 1.18.0 release? > > > > Regards, > > Kirti Dhar > > > > *From:* Hang Ruan > *Sent:* 07 June 2023 07:34 > *To:* Kirti Dhar Upadhyay K > *Cc:* user@flink.apache.o

Re: Custom Counter on Flink File Source

ext having metricGroup() as below- > > > > > > @Override > > *public* SplitEnumeratorMetricGroup metricGroup() { > > *return* *null*; > > } > > > > Am I doing any mistake? > > > > Regards, > > Kirti Dhar > > > &g

Re: flink14 sql sink kafka error

Hi, 湘晗刚, This error seem to be an error from the Kafka server. Maybe you should check whether the Kafka server occurs some error. Or you could provide more messages about the request. These information is too short to analyze, Best, Hang 湘晗刚 <1016465...@qq.com> 于2023年6月5日周一 15:08写道： >

Re: Custom Counter on Flink File Source

Hi, Kirti Dhar Upadhyay K. We could get the metric group from the context, like `SourceReaderContext` and `SplitEnumeratorContext`. These contexts could be found when creating readers and enumerators. See `AbstractFileSource#createReader` and `AbstractFileSource#createEnumerator`. Best, Hang

Re: SupportsReadingMetadata flow between table transformations

Hi, Yuval. `SupportsReadingMetadata` make connectors be able to append metadata columns to the schema of table sources. But one operator does not know the columns map to which column in the source. We could try to generate this information when parsing Flink sql. But it is hard for DataStream

Re: Reading KafkaSource state from a savepoint using the State Processor API

Hi, Charles, I am used to read the state in the debug mode. I always set the breakpoint at the return statemnet in `SavepointReader#read`. Then I could find the state I need in the field `SavepointMetadataV2 savepointMetadata`. Finally I could deserialize the state bytes with

Re: flink1.16.1 jdbc-3.1.0-1.16 There is a problem trying left join

Hi, yangxueyong, The filter(where condition) will be pushed down to the source if the connector implements the interface `SupportsFilterPushDown`. In your case, the sql planner analyzed that the records sent by `test_flink_res1` would satisfy the conditions (`name` = 'abc0.11317691217472489') and

Re: StatsdMetricsReporter is emitting all metric types as gauges

e gauge rather than giving its > current value. > > This means that counters are not monotonic and work like deltas that are > aggregated on the server side. > > Cheers, Iris. > > -- > > *Iris Grace Endozo,* Senior Software Engineer > *M *+61 435 108 697 > *E*

Re: Flink Job Restarts if the metadata already exists for some Checkpoint

Hi, amenreet, As Hangxiang said, we should use a new checkpoint dir if the new job has the same jobId as the old one . Or else you should not use a fixed jobId and the checkpoint dir will not conflict. Best, Hang Hangxiang Yu 于2023年5月10日周三 10:35写道： > Hi, > I guess you used a fixed JOB_ID, and

Re: StatsdMetricsReporter is emitting all metric types as gauges

Hi, Iris, The metrics have already be calculated in Flink. So we only need to report these metric as the gauges. For example, the metric `metricA` is a Flink counter and is increased from 1 to 2. The statsd gauge will be 2 now. If we register it as a statsd counter, we will send 1 and 2 to the

Re: OffsetCommitMode.Kafka_periodic with checkpointing enabled

Hi, Pritam, I see Martijn has responsed the ticket. Kafka source (FLIP-27) will commit offsets in two places: kafka consumer auto commit and invoke `consumer.commitAsync` when checkpoint is completed. - If the checkpoint is enabled and commit.offsets.on.checkpoint = true, kafka connector commits

Re: Flink Sql erroring at runtime

Hi, neha, I think the error occurred because of the deserialization. Is there some example data and runnable SQLs to reproduce the problem? Best, Hang neha goyal 于2023年5月2日周二 16:33写道： > Hello, > > I am using Flink 1.16.1 and observing a different behavior from Flink > 1.13.6. > > SELECT

Re: Query on flink-parquet

Hi, Anuj, Classes annotated with @Internal or @Experimental can be changed across any two releases. Classes annotated with @PublicEvolving only can be changed across minor releases (1.17.0 and 1.18.0). So the classes you mentioned may be changed. If the API changed in a new release, you have to

Re: Can flink1.15.2 use flink sql to create a broadcast table? Ididn't find anything related in https://flink.apache.org/

ottleneck. I think we can also read redis, but you still have > Any other options? > > 在 2023年5月6日 17:54，Hang Ruan 写道： > > Hi, yxy, > > I think this scenario could be resolved by a lookup join or a UDF. We can > store the ratio in the mysql table. Then we could read it by a look

Re: Can flink1.15.2 use flink sql to create a broadcast table? I didn't find anything related in https://flink.apache.org/

Hi, yxy, I think this scenario could be resolved by a lookup join or a UDF. We can store the ratio in the mysql table. Then we could read it by a lookup join or implement a UDF to read the ratio. Best, Hang yxy 于2023年5月6日周六 15:14写道： > Hello, we have a business scenario. We have a real-time

Re: Issue when Running the flink-kuberenetes-operator sql-runner (The file STDOUT does not exist on the TaskExecutor)

Hi, Nathan, I think the previous discussion[1] is helpful for you. Best, Hang [1] https://lists.apache.org/thread/cgwsr6f1l3202ktwcvtyxtdsgj2vyms1 Nathan Moderwell 于2023年5月5日周五 03:54写道： > Hi, > > I'm running the sql-runner example in the flink-kuberenetes-operator repo > and hitting an

Re: Apache Flink Kubernetes Operator 1.4.0

Hi, rania, I think the quick start document[1] is helpful for you. Other information could be found in its documents[2]. Best, Hang [1] https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/ [2]

Re: Why my flink sql job on yarn keep crash

Hi, Si-li, I think maybe it is not the root cause. You should find whether there are more exceptions in the JM log and TM logs. Best, Hang Shammon FY 于2023年4月18日周二 09:02写道： > Hi Si-li > > Could you give some more detailed exceptions? Or you can check the metrics > of your job such as memory

Re: 退订

退订请发送任意邮件到 user-unsubscr...@flink.apache.org，可以参考 https://flink.apache.org/community.html#how-to-subscribe-to-a-mailing-list Best, Hang 于2023年4月15日周六 11:57写道： > 退订 > > > > > -- > > 发自新浪邮箱客户端 >

Re: java.lang.RuntimeException: Error while getting state

Hi, igyu, It seems like the state in the join sql can not be recovered rightly. Do you change the columns in the join sql? If so, I think this may cause failing to recover from the checkpoint. Best, Hang igyu 于2023年4月14日周五 16:13写道： > I have a flink-SQL task. (enable savepoint) > I want change

Re: Support of CSV to AVRO Converter in DataStream FileSource

Hi, Kirti, I think you need a custom convertor for your csv files. The convertors provided by Flink only define how to translate the data into a Flink type. Best, Hang Kirti Dhar Upadhyay K via user 于2023年4月14日周五 15:27写道： > Hi Community, > > > > I am reading CSV data using data stream file

Re: File compression with File sink

Hi, Anuj, I searched the jira and found the related issue[1]. But this issue is still open now. Best, Hang [1] https://issues.apache.org/jira/browse/FLINK-6185 Anuj Jain 于2023年4月14日周五 14:58写道： > Hi Community, > > Does Flink File Sink support compression of output files, to reduce the > file

Re: Flink Job across Data Centers

Hi, Chirag, I am not sure whether this FLIP-268[1] is what you want. Best, Hang [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-268%3A+Kafka+Rack+Awareness Andrew Otto 于2023年4月12日周三 22:12写道： > Hi, I asked a similar question in this thread >

Re: Flink job manager conditional start of flink jobs

Hi, nage, I agree to the Shammon's suggestion after reading the context. Maybe you need a 'Job Management Service' to manage all jobs among different namespaces. I think the job management is not suitable for implementation in the Flink engine. Best, Hang Shammon FY 于2023年4月13日周四 11:34写道： >

Re: Quick question about flink document.

Hi, Dongwoo, I think there is no problem in this part. This part describes snapshotting Operator State, which is checkpointing. The checkpoint will store by the JobManager and use the checkpoint storage. Best, Hang Feng Jin 于2023年4月10日周一 00:32写道： > Hi Dongwoo > > > This can be quite

Re: How to set reblance in Flink Sql like Streaming api?

Hi, hiw, IMO, I think the parallelism 1 is enough for you job if we do not consider the sink. I do not know why you need set the lookup join operator's parallelism to 6. The SQL planner will help us to decide the type of the edge and we can not change it. Maybe you could share the Execution graph

Re: Access ExecutionConfig from new Source and Sink API

Hi, christopher, I think there is already about the ExecutionConfig for new Sink API in the FLIP-287[1]. What we actually need is a read-only ExecutionConfig for Source API and Sink API. Maybe we could continue to discuss this topic under FLIP-287. Best, Hang [1]

Re: org.apache.flink.table.api.ValidationException

Hi, This error occurs when the data type can not be parsed. You could read this part to see more details about the User-Defined Data Types[1]. Best, Hang [1] https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/types/#user-defined-data-types 柒朵 <1303809...@qq.com>

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

Congratulations! Best, Hang yu zelin 于2023年3月28日周二 10:27写道： > Congratulations! > > Best, > Yu Zelin > > 2023年3月27日 17:23，Yu Li 写道： > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project

  1   2   >