Re: Error about "Rejected TaskExecutor registration at the ResourceManger"

Hi Jing, Thank you for your reply, that cluster is terminated and will provide the log if it occurs again. On Wed, Jun 2, 2021 at 11:17 AM JING ZHANG wrote: > Hi Kai, > The reason why job job cannot be recovered maybe not directly related to > the exception you mentioned in your email. > Would

Re: flink1.11.2 yarn-session 部分类路径未加载

启动是没有问题的，问题在于sql-client 使用yarn-session时会报错误。 /opt/flink-1.11.2/bin/sql-client.sh embedded -s yarn-session 启动sql-client之后执行hive数据查询报错，报错如下： [ERROR] Could not execute SQL statement. Reason: org.apache.flink.runtime.rest.util.RestClientException: [Internal server error., SinkConversionToRow ->

回复：flink1.12版本，yarn-application模式Flink web ui看不到日志

你这个解决了吗，我也遇到了同样的问题 -- 原始邮件 -- 发件人: todd http://apache-flink.147419.n8.nabble.com/

Re: flink1.11.2 yarn-session 部分类路径未加载

不需要 mapreduce 相关库吧。 我看我的 job 里加载到 classpath 的也没有 mapreduce。 Zhiwen Sun On Wed, Jun 2, 2021 at 11:56 AM datayangl wrote: > flink1.11.2 启动yarn-session之后发现，有部分类路径始终没有加载到class_path中去 > 环境变量配置如下： > < > http://apache-flink.147419.n8.nabble.com/file/t919/66604010-2A08-4A68-8478-70A27D61224B.png> >

flink1.11.2 yarn-session 部分类路径未加载

flink1.11.2 启动yarn-session之后发现，有部分类路径始终没有加载到class_path中去 环境变量配置如下： 其中tm的日志如下： tm.log 其中hadoop-mapreduce-client相关的类路径一直没有加载到class_path中，求指教

退订

Re: Error about "Rejected TaskExecutor registration at the ResourceManger"

Hi Kai, The reason why job job cannot be recovered maybe not directly related to the exception you mentioned in your email. Would you like provide complete jobmanager.log and taskmanager.log. Maybe we could find some hints there. Best regards, JING ZHANG Kai Fu 于2021年6月2日周三 上午7:23写道： > HI

退订

退订

Re: Flink Sql 的/checkpoint/shared/文件夹大小不断增长，源数据没有数据激增，应该如何控制？

那会一直增大下去吗，我跑了4天，ckp一直变大，没有稳定的迹象。是不是我需要调整compaction的配置 -- Sent from: http://apache-flink.147419.n8.nabble.com/

Re: Flink是否支持自定义的限流功能

是的，因为我们是写ES，ES不支持动态反压，所以想问下有没有可配置的静态限流的方法 -- Sent from: http://apache-flink.147419.n8.nabble.com/

退订

Re: Error about "Rejected TaskExecutor registration at the ResourceManger"

HI Till, Thank you for your response, per my observation that the process lasted for ~1 day, and cannot be recovered and we killed the cluster finally. On Tue, Jun 1, 2021 at 9:47 PM Till Rohrmann wrote: > Hi Kai, > > The rejection you are seeing should not be serious. The way this can >

Re: Does WatermarkStrategy.withIdleness work?

JFYI in case other users find this in the future. ProcessingTimeTrailingBoundedOutOfOrdernessTimestampExtractor has a small issue if modified to be used with the new watermark API and if the events can have the same timestamp. I changed my code to do this onPeriodicEmit. In this situation, we

Re: How to check is Kafka partition "idle" in emitRecordsWithTimestamps

Hi Till, >However, this will stall the whole reading process if there is a partition >which has no more data. Hence, you will probably also need a mechanism to >advance the watermark if the partition becomes idle. This is why I need to find out is partition idle. Looks like Kafka Flink

Re: Best practice for adding support for Kafka variants

Thank you, Roman. I should have said our own flavor of Kafka and not version. Thanks for the reference of the new source and sink interfaces, though, as it seems like the interfaces we should be implementing to use our custom Kafka connector. I did notice however that the FLIP does not cover

Re: Flink Metrics Naming

Some more background on MetricGroups: Internally there (mostly) 3 types of metric groups: On the one hand we have the ComponentMetricGroups (like TaskManagerMetricGroup) that describe a high-level Flink entity, which just add a constant expression to the logical scope(like taskmanager, task

Re: Reading Flink states from svaepoint uning State Processor API

Hi Min, The only requirement is that your state descriptors be configured identically as those used in your datastream API. So if you registered custom TypeInformation / serializer in your streaming job you will need those here as well. I would also look at the ExecutionConfig on your DataStream

Re: S3 + Parquet credentials issue

Hi Angelo, what Svend has written is very good advice. Additionally, you could give us a bit more context by posting the exact stack trace and the exact configuration you use to deploy the Flink cluster. To me this looks like a configuration/setup problem in combination with AWS. Cheers, Till

Re: Reading Flink states from svaepoint uning State Processor API

Hi Min, Usually, you should be able to provide type information and thereby a serializer via the StateDescriptors which you create to access the state. If this is not working, then you need to give us a bit more context to understand what's not working. I am also pulling in Seth who is the

Re: savepoint fail

Responded as part of the following discussion https://lists.apache.org/x/thread.html/re85a1fb3f17b5ef1af2844fc1a07b6d2f5e6237bf4c33059e13890ee@%3Cuser.flink.apache.org%3E. Let's continue the discussion there. Cheers, Till On Mon, May 31, 2021 at 11:02 AM 周瑞 wrote: > HI: > When

Re: Flink Metrics Naming

Upon further inspection, it seems like the user scope is not universal (i.e. comes through the connectors and not UDFs (like rich map function)), but the question still stands if the process makes sense. > On Jun 1, 2021, at 10:38 AM, Mason Chen wrote: > > Makes sense. We are primarily

Re: classloader.resolve-order is not honored when submitting job to a remote cluster

Hi Till, The PR covers the problem and it will fix the inconsistent class loading issue On Tue, Jun 1, 2021 at 9:55 PM Till Rohrmann wrote: > Hi Tao, > > I think this looks like a bug to me. Could it be that this problem is > covered by [1, 2]? Maybe you can review this PR and check whether it

Re: Flink Metrics Naming

Makes sense. We are primarily concerned with removing the metric labels from the names as the user metrics get too long. i.e. the groups from `addGroup` are concatenated in the metric name. Do you think there would be any issues with removing the group information in the metric name and

Re: RabbitMQ source does not stop unless message arrives in queue

Hi all, Apologies for not following up sooner. Thank you Austin for creating FLINK-22698. It seems that the issue is well understood and a fix is currently under development/review. Please let me know if there is anything additional that I can do. I look forward to testing out a new version of

Re: Got exception when running the localhost cluster

Hi Lingfeng, Youngwoo is right. Flink currently officially supports Java 8 and Java 11. Cheers, Till On Mon, May 31, 2021 at 9:00 AM Youngwoo Kim (김영우) wrote: > Hi Lingfeng, > > I believe Java 8 or 11 is appropriate for the Flink cluster at this point. > I'm not sure that Flink 1.13 supports

Re: Flink Metrics Naming

The uniqueness of metrics and the naming of the Prometheus reporter are somewhat related but also somewhat orthogonal. Prometheus works similar to JMX in that the metric name (e.g., taskmanager.job.task.operator.numRecordsIn) is more or less a _class_ of metrics, with tags/labels allowing you

Re: Flink kafka

Responded as part of the following discussion https://lists.apache.org/x/thread.html/re85a1fb3f17b5ef1af2844fc1a07b6d2f5e6237bf4c33059e13890ee@%3Cuser.flink.apache.org%3E. Let's continue the discussion there. Cheers, Till On Sun, May 30, 2021 at 2:32 PM 周瑞 wrote: > > 程序用于测试 flink kafka exactly

Re: classloader.resolve-order is not honored when submitting job to a remote cluster

Hi Tao, I think this looks like a bug to me. Could it be that this problem is covered by [1, 2]? Maybe you can review this PR and check whether it solves the problem. If yes, then let's quickly get it in. [1] https://issues.apache.org/jira/browse/FLINK-21445 [2]

Re: Error about "Rejected TaskExecutor registration at the ResourceManger"

Hi Kai, The rejection you are seeing should not be serious. The way this can happen is the following: If Yarn restarts the application master, Flink will try to recover previously started containers. If this is not possible or Yarn only tells about a subset of the previously allocated containers,

Re: Getting Abstract method Error with flink 1.13.0 and 1.12.0

It appears as if flink-siddhi was build against Flink 1.9.0, so it may just not be compatible with later versions of Flink. You will either need to reach out to the maintainers of flink-siddhi, or migrate it to a later Flink version yourself. On 6/1/2021 3:11 PM, Dipanjan Mazumder wrote:

Re: Flink Metrics Naming

Hi Mason, The idea is that a metric is not uniquely identified by its name alone but instead by its path. The groups in which it is defined specify this path (similar to directories). That's why it is valid to specify two metrics with the same name if they reside in different groups. I think

Re: How to check is Kafka partition "idle" in emitRecordsWithTimestamps

Hi Alexey, looking at KafkaTopicPartitionStatus, it looks that it does not contain this information. In a nutshell, what you probably have to do is to aggregate the watermarks across all partitions and then pause the consumption of a partition if its watermark advances too much wrt to the minimum

Re: recover from svaepoint

The error message says that we are trying to reuse a transaction id that is currently being used or has expired. I am not 100% sure how this can happen. My suspicion is that you have resumed a job multiple times from the same savepoint. Have you checked that there is no other job which has been

Re: Getting Abstract method Error with flink 1.13.0 and 1.12.0

Hi ,    I have integrated flink-siddhi library ([com.github.haoch/flink-siddhi_2.11 "0.2.2-SNAPSHOT"]) and i tried to configure and implement control stream from flink-siddh and it broke with AbstractMethodError. When i tried running the same with flink 1.11.0 it worked. More Details is

Re: Getting Abstract method Error with flink 1.13.0 and 1.12.0

Thanks Till will do so ... On Tuesday, June 1, 2021, 06:22:32 PM GMT+5:30, Till Rohrmann wrote: Hi Dipanjan, this type of question is best sent to Flink's user mailing list because there are a lot more people using Flink who could help you. The dev mailing list is intended to be used

Re: Getting Abstract method Error with flink 1.13.0 and 1.12.0

Hi Dipanjan, this type of question is best sent to Flink's user mailing list because there are a lot more people using Flink who could help you. The dev mailing list is intended to be used for development discussions. Cheers, Till On Tue, Jun 1, 2021 at 1:31 PM Dipanjan Mazumder wrote: > Hi ,

回复：窗口函数使用的时间类型

谢谢！明天回公司再调试。 ---原始邮件--- 发件人: "MOBIN"<18814118...@163.com 发送时间: 2021年6月1日(周二) 晚上7:41 收件人: "user-zh@flink.apache.org"https://help.aliyun.com/document_detail/62512.html?spm=a2c4g.11186623.6.827.49531b09XfgsU7 | | MOBIN | 签名由网易邮箱大师定制 在2021年06月1日 19:37，guoyb<861277...@qq.com 写道： 好的，谢谢！ 我试试

????

回复：窗口函数使用的时间类型

其次可以参考下阿里的demo: https://help.aliyun.com/document_detail/62512.html?spm=a2c4g.11186623.6.827.49531b09XfgsU7 | | MOBIN | 签名由网易邮箱大师定制 在2021年06月1日 19:37，guoyb<861277...@qq.com> 写道： 好的，谢谢！ 我试试 ---原始邮件--- 发件人: "Shuo Cheng"

回复：窗口函数使用的时间类型

好的，谢谢！ 我试试 ---原始邮件--- 发件人: "Shuo Cheng"

Re: 窗口函数使用的时间类型

SQL 流作业 window 可定义在两种时间属性类型字段上: 1) event time: ddl 中需要给时间类型字段 (timestamp) 定义 watermark 2) process time: 使用 PROCTIME() On 6/1/21, guoyb <861277...@qq.com> wrote: > 是的。 > > > 大神能否指条明路解决这问题。 > > > > ---原始邮件--- > 发件人: "MOBIN"<18814118...@163.com > 发送时间: 2021年6月1日(周二) 晚上7:09 > 收件人:

?????? Pyflink jdbc????

?? ---- ??: "user-zh"

Re: Pyflink jdbc相关

这样试试，把”\”改成”/“： file:///D:/Pyproject/flink-connector-jdbc_2.11-1.13.1.jar > 2021年6月1日 下午5:40，琴师 <1129656...@qq.com> 写道： > > 再请叫一个问题，我在pycharm使用时候引用windows的地址不能引用，比如file:///D:\Pyproject\flink-connector-jdbc_2.11-1.13.1.jar; > 这样不能引入，大佬有用用过吗？ > > > --原始邮件-- >

回复：窗口函数使用的时间类型

是的。 大神能否指条明路解决这问题。 ---原始邮件--- 发件人: "MOBIN"<18814118...@163.com 发送时间: 2021年6月1日(周二) 晚上7:09 收件人: "user-zh@flink.apache.org"

回复：窗口函数使用的时间类型

是不是报的类似下面的错？ Window aggregate can only be defined over a time attribute column, but TIMESTAMP(3) encountered | | MOBIN | 签名由网易邮箱大师定制 在2021年06月1日 19:00，guoyb<861277...@qq.com> 写道： tumble() 开窗，需要的事件时间到底需要什么时间类型？一直报时间不对 timestamp(3) datetime time 都试过了，没有一个对的。

窗口函数使用的时间类型

tumble() 开窗，需要的事件时间到底需要什么时间类型？一直报时间不对 timestamp(3) datetime time 都试过了，没有一个对的。

Re: How to use or configure flink checkpointing with siddhi internal state

Hi Till,     Thanks so that means it should work will try and see .. RegardsDipanjan On Tuesday, June 1, 2021, 01:48:19 PM GMT+5:30, Till Rohrmann wrote: Hi Dipanjan, I am assuming that you are using the flink-siddhi library [1]. I am not an expert but it looks as if the

Re: kafka exactly-once语义下,从svaepoint恢复报错

'properties.transaction.timeout.ms' = '3' 配置的太短了，30s transactionalId 就过期了。 估计 都来不去启动吧 官网的原文 https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/ Semantic.EXACTLY_ONCE mode relies on the ability to commit transactions that were started before

Re: Flink是否支持自定义的限流功能

flink 的反压机制 不就是在限流么？ suisuimu <726400...@qq.com> 于2021年6月1日周二 下午5:37写道： > Flink从Kafka读取数据时，是否支持用户自定义的限流策略。 > 例如根据消息中的某个字段的名称，设置流控规则。 > 请问是否支持呢？还是需要自己借助第三方组件（例如sentinel）来实现？ > > > > -- > Sent from: http://apache-flink.147419.n8.nabble.com/ -- Best， pp

?????? Pyflink jdbc????

pycharmwindowsfile:///D:\Pyproject\flink-connector-jdbc_2.11-1.13.1.jar; ?? ---- ??:

Re: 不同的程序在同一时间段报同一个异常

你的网络环境是怎么样？ 是在docker 上跑么？还是怎么？ 从报错上看，netty 无法解码导致的，但是为什么会出现这样的现象？ 或许 你可以把问题贴的在详细一点 5599 <673313...@qq.com> 于2021年6月1日周二 下午2:32写道： > 退订 > > > > > --原始邮件-- > 发件人: "r pp" 发送时间: 2021年6月1日(星期二) 下午2:07 > 收件人: "user-zh" 主题: Re: 不同的程序在同一时间段报同一个异常 > > > > 你的程序有挂掉么？ > > mq

Flink是否支持自定义的限流功能

Flink从Kafka读取数据时，是否支持用户自定义的限流策略。 例如根据消息中的某个字段的名称，设置流控规则。 请问是否支持呢？还是需要自己借助第三方组件（例如sentinel）来实现？ -- Sent from: http://apache-flink.147419.n8.nabble.com/

自定义带有状态的udf

自定义UDF 实现CheckpointedFunction 伪代码如下 发现并没有执行initializeState public class ClusterInfoCollectUdf extends ScalarFunction implements CheckpointedFunction { private static final LoggerLOGGER = LoggerFactory.getLogger(ClusterInfoCollectUdf.class); private transient

?????? Pyflink jdbc????

2.11?? ---- ??: "user-zh"

如何自定义带有状态的UDF

自定义UDF 实现CheckpointedFunction 伪代码如下 发现并没有执行initializeState public class ClusterInfoCollectUdf extends ScalarFunction implements CheckpointedFunction { private static final LoggerLOGGER = LoggerFactory.getLogger(ClusterInfoCollectUdf.class); private transient

Re: Pyflink jdbc相关

Hi， 本地执行： 1）用flink-connector-jdbc_2.11-1.13.1.jar试试？因为PyFlink里默认待的JAR包是scala 2.11的 flink run: 1) 你注册的sink表的名字为“print”，不是”table_sink”，但是在SQL语句里用的table_sink。 > 2021年6月1日 下午4:33，琴师 <1129656...@qq.com> 写道： > > Hi， > 我按着微信分享https://mp.weixin.qq.com/s/GyFTjQl6ch8jc733mpCP7Q >

Re: Flink Sql 的/checkpoint/shared/文件夹大小不断增长，源数据没有数据激增，应该如何控制？

Hi， 增量checkpoint上传的是sst文件本身，里面可能有一部分空间是被无用数据占据的，你可以理解成增量checkpoint上传的是受到空间放大影响的RocksDB的数据，如果因为单机的数据量较小，没有及时触发compaction的话，确实存在整个远程checkpoint目录数据大于当前实际空间的情况。而关闭增量checkpoint，上传的其实是与savepoint格式一样的kv数据对，Flink会遍历整个DB，将目前有效的数据写出到远程。所以你关闭增量checkpoint，而发现checkpoint目录保持恒定大小的话，说明真实有效数据的空间是稳定的。

Pyflink jdbc????

Hi?? ??https://mp.weixin.qq.com/s/GyFTjQl6ch8jc733mpCP7Q pyflink??jdbc??jar??jdbc??flink??1.13.1 from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import StreamTableEnvironment,

flink sql1.13.1????change log??????join????????????????????????????

|insert into dwd_order_detail |select | ord.Id, | ord.Code, | Status | concat(cast(ord.Id as String),if(oed.Id is null,'oed_null',cast(oed.Id as STRING)),DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd')) as uuids, | TO_DATE(DATE_FORMAT(LOCALTIMESTAMP,'-MM-dd')) as

Re: How to use or configure flink checkpointing with siddhi internal state

Hi Dipanjan, I am assuming that you are using the flink-siddhi library [1]. I am not an expert but it looks as if the AbstractSiddhiOperator overrides the snapshotState [2] method to store the Siddhi state in Flink. [1] https://github.com/haoch/flink-siddhi [2]

????

kafka exactly-once语义下,从svaepoint恢复报错

您好：kafka在exactly-once语义下,从svaepoint恢复报错。初步排查认为是kafka事务使用了旧的epoch。请问这个问题怎么处理？ //todo 通过配置传进来 env.setParallelism(1); env.enableCheckpointing(60L, CheckpointingMode.EXACTLY_ONCE); // checkpoint的清除策略(即使任务被显示地取消也会保留checkpoint) env.getCheckpointConfig()

?????? ??????????????????????????????????

---- ??: "r pp"

Re: 不同的程序在同一时间段报同一个异常

有些程序挂掉，有些没有 r pp 于2021年6月1日周二 下午2:07写道： > 你的程序有挂掉么？ > > mq sun 于2021年5月31日周一 下午7:23写道： > > > 大家好： > > 最近在生产中，不同项目组的两个flink程序在同一时间段都报下面异常 > > ERROR org.apache.flink.runtime.blob.BlobServerConnection -Error while > > excuting Blob connection > > . > > . > > . > > > > >

Re: 不同的程序在同一时间段报同一个异常

你的程序有挂掉么？ mq sun 于2021年5月31日周一 下午7:23写道： > 大家好： > 最近在生产中，不同项目组的两个flink程序在同一时间段都报下面异常 > ERROR org.apache.flink.runtime.blob.BlobServerConnection -Error while > excuting Blob connection > . > . > . > > org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException >