Flink-Iceberg Table Sink failing with org.apache.hadoop.fs.s3a.S3AStorageStatistics Cast exception

Hi We are running a Flink job that delivers Kafka data to an Iceberg table. The job uses the *org.apache.iceberg.flink.CatalogLoader* and *org.apache.iceberg.flink.TableLoader *interfaces in combination with *org.apache.iceberg.flink.sink.FlinkSink *where the catalog type is Hive. We have had

Re: [EXT] Vertica jdbc sink error

Hi Martin, here is a Jira ticket I created: https://issues.apache.org/jira/browse/FLINK-27429 I guess you now assign it to me, right? :) Best Regards, Jasmin On 19.04.2022., at 22:34, Jasmin Redzepovic mailto:jasmin.redzepo...@superbet.com>> wrote: Hi Martin, Thanks for your answer.

Re: AvroRowDeserializationSchema

Yes, I think so~ On Thu, Apr 28, 2022 at 11:00 AM lan tran wrote: > Hi Dian, > > Sorry for missing your mail, so if I did as your suggestion and the Flink > somehow crashed and we have to restart the service, does the Flink job know > the offset where does it read from Kafka ? > > > > Sent from

Temporal join fails with "unexpected correlate variable $cor0 in the plan"

Hi all, I'm trying to join the following two tables using a temporal join: *table_1* ( `f0` STRING NOT NULL, `f1` DOUBLE NOT NULL, `rowtime` TIMESTAMP(3) METADATA, WATERMARK FOR `rowtime`: TIMESTAMP(3) AS rowtime - INTERVAL '10' SECOND ) *table_2:* ( `f0` STRING NOT NULL, `f1`

Using the official flink operator and kubernetes secrets

Hi all, I'm trying to use a kubernetes secret as a command line argument in my job and the text replacement doesn't seem to be happening. I've verified passing the custom args via the command line on my local flink cluster but can't seem to get the environment var replacement to work.

RE: AvroRowDeserializationSchema

Don’t expect that answer =))However, I am very appreciate everything you did Thanks again for helping me out.Best,Quynh. Sent from Mail for Windows From: Dian FuSent: Thursday, April 28, 2022 2:59 PMTo: lan tranCc: user@flink.apache.orgSubject: Re: AvroRowDeserializationSchema Yes, I think so~ On

Unsubscribe

Unsubscribe Regards, Bhawana

Re: Unsubscribe

To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org . > 2022年4月27日 下午12:03，Amit Bhatia 写道： > > Unsubscribe > > Regards, > Amit Bhatia

Re: Unsubscribe

To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org . > 2022年4月28日 下午6:45，bhawana gupta 写道： > > Unsubscribe > > Regards, > Bhawana

Re: How to debug Metaspace exception?

I think what I meant was "either add it to /lib, or [if it is already in /lib but also bundled in the jar] add it to the parent-first patterns." On 28/04/2022 15:56, Chesnay Schepler wrote: Pretty sure, even though I seemingly documented it incorrectly :) On 28/04/2022 15:49, John Smith

Re: How to debug Metaspace exception?

You sure? - *JDBC*: JDBC drivers leak references outside the user code classloader. To ensure that these classes are only loaded once you should either add the driver jars to Flink’s lib/ folder, or add the driver classes to the list of parent-first loaded class via

Re: How to debug Metaspace exception?

Pretty sure, even though I seemingly documented it incorrectly :) On 28/04/2022 15:49, John Smith wrote: You sure? * /JDBC/: JDBC drivers leak references outside the user code classloader. To ensure that these classes are only loaded once you should either add the driver jars to

Re: Using the official flink operator and kubernetes secrets

Also, just declaring it in the flink configs should be sufficient, no need to define it in the pod templates: flinkConfiguration: kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" Best, Matyas On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás wrote: >

Re: Using the official flink operator and kubernetes secrets

Hi Francis, I suggest accessing the environment variables directly, no need to pass them as command arguments I guess. Best, Matyas On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy < francis.con...@switchdin.com> wrote: > Hi all, > > I'm trying to use a kubernetes secret as a command line

Re:Temporal join fails with "unexpected correlate variable $cor0 in the plan"

Hi, can you provide the version of the Flink with this exception? I test your SQL in Flink/master and it works. My test SQL is the following. create table table_1 ( `f0` STRING NOT NULL, `f1` DOUBLE NOT NULL, `rowtime` TIMESTAMP(3), WATERMARK FOR `rowtime` AS rowtime - INTERVAL '10' SECOND )

Re: How to debug Metaspace exception?

Ok so to summarize... - Build my job jar and have the JDBC driver as a compile only dependency and copy the JDBC driver to flink lib folder. Or - Build my job jar and include JDBC driver in the shadow, plus copy the JDBC driver in the flink lib folder, plus make an entry in config for

Re: How to debug Metaspace exception?

I assume you will take action on your side to track and fix the doc? :) On Thu, Apr 28, 2022 at 11:12 AM John Smith wrote: > Ok so to summarize... > > - Build my job jar and have the JDBC driver as a compile only > dependency and copy the JDBC driver to flink lib folder. > > Or > > - Build my

Re:How to dynamically modify the schema information of a table

I tried this and found that the comment information will be lost forever when using "fromDataStream". I think you can create a issue to report this bug. At 2022-04-22 10:44:47, "草莓" wrote: The following is the Java code @Test public void test(){ StreamExecutionEnvironment env =

Broadcast State + Stateful Operator + Async IO

Hello, My application has a stateful operator which leverages RocksDB to store a large amount of state. It, along with other operators receive configuration as a broadcast stream (KeyedBroadcastProcessFunction). The operator depends upon another input stream that triggers some communication with

RE: Flink-Iceberg Table Sink failing with org.apache.hadoop.fs.s3a.S3AStorageStatistics Cast exception

I solved this problem by removing the Hadoop classpath from Flink cluster deployment. On 2022/04/28 09:04:50 Terry Heathcote wrote: > Hi > > We are running a Flink job that delivers Kafka data to an Iceberg table. > The job uses the *org.apache.iceberg.flink.CatalogLoader* and >

Re: Flink team staffing

Thousands of events per second depending on their size and complexity isn’t a big overhead. If you’re in AWS you might want to look at Kinesis Data Analytics. On Wed, Apr 27, 2022 at 19:32, Wei Liu wrote: > Hi everyone, > > I've been thinking about running some production-critical applications

Checkpoint Timeout Troubleshooting

Hello, I am running into checkpoint timeouts and am looking for guidance on troubleshooting. What should I be looking at? What configuration parameters would affect this? I am afraid I am a Flink newbie so I am still picking up the concepts. Additional notes are below, anything else I can

Re: Broadcast State + Stateful Operator + Async IO

Hi Vishal I want to understand your needs first. Your requirements are: After a stateful operator receives a notification, it needs to traverse all the data stored in the operator state, communicate with an external system during the traversal process (maybe similar to join?). In order to improve

Re: AvroRowDeserializationSchema

Oh, I just missed your last question, sorry for that. The offset is stored in the checkpoint and it will recover the offset from the checkpoint when the job failover. Things which you may need to pay attention to: 1) Enable the checkpoint and configure it if necessary [1] 2) Specify the start up

Re: Checkpoint Timeout Troubleshooting

Hi Sam I think the first step is to see which part of your Flink APP is blocking the completion of Checkpoint. Specifically, you can refer to the "Checkpoint Details" section of the document [1]. Using these methods, you should be able to observe where the checkpoint is blocked, for example, it

Re: NO jobmanager.log for yarn-per-job submitted with sql-client.sh

Hi ruiyun, Is there any log4j configuration file or logback.xml in your Flink configuration directory? Yarn should check whether the file exists first and set relative log options. > On 26 Apr 2022, at 5:21 PM, ruiyun wan wrote: > > Flink Version (1.13) > There is not jobmanager.log file

Re: How to dynamically modify the schema information of a table

Thank you for your reporting. I created a ticket in JIRA. You can track this problem in this link: https://issues.apache.org/jira/browse/FLINK-27449 . > On 22 Apr 2022, at 10:44 AM, 草莓 wrote: > > The following is the Java code > > @Test >

Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

Hi Shengkai, Thank you for the reply. The UDF getEventTimeInNS uses timestamps of both streamA and streamB to calculate the true event time for streamB events. For illustrating purpose, we can consider it to be like this: public Long eval( Long baseTimeStampFromA, Long

Flink native kubernetes 下没有足够的 slot 问题

您好， 我最近在尝试使用 native kubernetes 方式部署 flink 集群。我根据官方文档使用 session 模式部署了一个集群，并在上面提交了一个并行度为 5 的测试 Job，参数设置了 -D-Dtaskmanager.numberOfTaskSlots=3。提交后可以在前端看到创建了两个 slot 数量为 3 的 Taskmanager，并且其中有一个 Taskmanager 显示 1 slot free。但是 Job 无法正常启动，得到了 has no more allocated slots for job

Re: 对Flink Table Store咨询

哈喽，感谢你的关注 Tison是对的，Table Store是一个库。我们目前并没有发布maven依赖。 目前你可以有两种方式来调试： 1.在Table Store的工程里调试 2.在拿到flink-table-store-dist下的 pre bundled jar，放到你工程的classpath下来调试 入口类的话是通过SQL的方式： TableEnvironment.executeSql("CREATE TABLE XX (...)"); 当classpath下包含 table store 的 jar 时，会服务发现到 TableStore 的 factory，进而走到

Re: Flink native kubernetes 下没有足够的 slot 问题

万分抱歉！邮件客户端误触发送出去了。。。 请问为什么 slot 总数充足的情况下会出现 slot 不足的异常？如何在 native kubernetes 下正确启动一个多并行度任务？ 感谢大家抽空看我的邮件！ Best, 潘 On Apr 28 2022, at 5:28 pm, Pan Junxun wrote: > 您好， > > 我最近在尝试使用 native kubernetes 方式部署 flink 集群。我根据官方文档使用 session > 模式部署了一个集群，并在上面提交了一个并行度为 5 的测试 Job，参数设置了 >

?????? ??Flink Table Store????

-- set root path to session config SET 'table-store.path' = '/tmp/table_store'; ??flink??config table sql ??demosql cli ?? ---- ??:

?????? ??Flink Table Store????

?? import org.apache.flink.configuration.Configuration; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.TableConfig; import org.apache.flink.table.api.TableEnvironment; public class sql { public static void main(String[] args) {

?????? ??Flink Table Store????

?? import org.apache.flink.configuration.Configuration; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.TableConfig; import org.apache.flink.table.api.TableEnvironment; public class sql { public static void main(String[] args) {

Re: Flink native kubernetes 下没有足够的 slot 问题

当 SlotManager 向 TaskExecutor 为作业申请 Slot 后，TaskExecutor 会向 JobMaster offer这些 Slots。从 TaskExecutor 接受到 SlotManager 的请求后会注册一个定时器，如果在定时器计时结束时仍然没有将 Slots offer 给 JobMaster，会触发这个问题。 Slot timeout 的时间配置项为taskmanager.slot.timeout，如果没有单独配置，则使用 akka.ask.timeout 的值（默认为 10s）。 可以尝试增加

?????? ??Flink Table Store????

?? -- -- ??: "user-zh"

Re: Flink native kubernetes 下没有足够的 slot 问题

Hi, Junxun 按照你的说法前半部分是符合预期的，并行度为 5 的作业需要 2 个 slot 数量为 3 的 TaskManager。 这里没看到具体的报错日志，方便提供下完成日志吗？以及对应的 flink 版本信息。 > 2022年4月28日 下午5:28，Pan Junxun 写道： > > 您好， > > 我最近在尝试使用 native kubernetes 方式部署 flink 集群。我根据官方文档使用 session > 模式部署了一个集群，并在上面提交了一个并行度为 5 的测试 Job，参数设置了 >

Re: Flink native kubernetes 下没有足够的 slot 问题

感谢回复！ 日志内容如下： 2022-04-28 19:58:20 org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138) at

Re: Flink native kubernetes 下没有足够的 slot 问题

忘记说了，Flink 版本是 1.13.1 On Apr 28 2022, at 8:04 pm, Pan Junxun wrote: > 感谢回复！ > > 日志内容如下： > > 2022-04-28 19:58:20 > org.apache.flink.runtime.JobException: Recovery is suppressed by > NoRestartBackoffTimeStrategy > at >

Re:Flink SQL??????Java code????debug

Hi??FlinkjaninoJanino[1]??org.codehaus.janino.source_debugging.enable=true??org.codehaus.janino.source_debugging.dir=mypathdebug[1] http://janino-compiler.github.io/janino/#debugging ?? 2022-04-25 17:04:30??"zhiyezou" <1530130...@qq.com.INVALID>

Re: flink 任务对接k8s的第三方jar包管理问题

你可以试试将第三方包放进你的user jar里，制作一个fat jar. 或者你保证每一个container都存在这个第三方包，不一定是在lib下。然后提交作业的时候使用pipeline.classpath去指定这个路径。 参考https://nightlies.apache.org/flink/flink-docs-master/zh/docs/dev/python/dependency_management/ > On 25 Apr 2022, at 6:21 PM, 天道酬勤 <1262420...@qq.com.INVALID> wrote: > >

Re: Flink native kubernetes 下没有足够的 slot 问题

感谢大佬指点，我换到 1.14.4 后这个问题解决了 On Apr 28 2022, at 9:03 pm, huweihua wrote: > 当 SlotManager 向 TaskExecutor 为作业申请 Slot 后，TaskExecutor 会向 JobMaster offer这些 > Slots。从 TaskExecutor 接受到 SlotManager 的请求后会注册一个定时器，如果在定时器计时结束时仍然没有将 Slots > offer 给 JobMaster，会触发这个问题。 > > Slot timeout