Re: State Processor API and existing state

Hi Marco, > I assume that all the data within the checkpoint are stored within the given Savepoint. Is that assumption correct? Yes > I have not figured out how to correct / augment / fix the state though. Can somebody please explain? Please try this way. 1. Load old savepoint file, create

Re: Savepoint failure with operation not found under key

Thanks for the context Chesnay. Yes, I sent both requests to the same JM. Best regards Rainie On Mon, Jun 28, 2021 at 8:33 AM Chesnay Schepler wrote: > Ordinarily this happens because the status request is sent to a different > JM than the one who received the request for creating a savepoint.

Re: 退订

如果需要取消订阅 u...@flink.apache.org 邮件组，请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 。 > 在 2021年6月29日，11:01，大雨 <95133...@qq.com.INVALID> 写道： > > 退订

Re: flink 1.13.0 中cumulate window 使用

是否insert应该和这个cumulate window没关系了。需要考虑你的test_out的定义，定primary key可以试一试。 此外，@Leonard Xu，我想知道window tvf，table values function，这么个词想表达啥呢？表值函数？ 其次，为啥语法中是 from table(CUMULATE(...)) 而不是直接 from CUMULATE(...)。此处的table()的这个table算是啥呢？函数？ 那么CUMULATE(...)返回的是啥呢，是某种类型，可以用于table函数的参数吗。 还是 table(xx)理解为类型转换呢？ 邹云鹤

回复: Re: How to convert local to iso 8601 time in flink sql?

Hi Xu I use Elasticsearch as Sink. If a timestamp field is sent into Elasticsearch without any timezone information, then it will be assumed to be UTC time (Coordinated Universal Time)[1]. [1] https://www.elastic.co/blog/converting-local-time-to-iso-8601-time-in-elasticsearch

[DISCUSS] Better user experience in the WindowAggregate upon Changelog (contains update message)

When WindowAggregate works upon Changelog which contains update messages, UPDATE BEFORE message may be dropped as a late message. [1] In order to handle late UB message, user needs to set *all* the following 3 parameters: (1) enable late fire by setting table.exec.emit.late-fire.enabled : true

Re: 退订

退订请发送到 user-zh-unsubscr...@flink.apache.org 而不是 user-zh@flink.apache.org Best, Jingsong On Mon, Jun 28, 2021 at 5:56 PM luoye <13033709...@163.com> wrote: > 退订 -- Best, Jingsong Lee

Re: Flink Sql 1.13 UDF ERROR

Hi, 你可以创建个JIRA，让Timo看看，UDAF引入了新的类型推导，可能有问题 Best, Jingsong On Tue, Jun 29, 2021 at 7:10 AM Roc Marshal wrote: > > > Hi, All. > > > 请教一个在最新的1.13.1 api升级调研中遇到的问题，谢谢大家： > > > 版本: 1.13.1 > 运行模式： IDE-application > --- > about udf define...

????

Re: How to convert local to iso 8601 time in flink sql?

Hi, Unfortunately Flink SQL doesn’t support TIMESTAMP WITH TIME ZONE type yet[1], maybe the you can try write an UDF to convert the timestamp '2021-06-29 09:00:00’ field to String(the string representation like '2021-06-29T09:00:00+08:00’). And could you share your scenario about using

Re:回复：flinksql写入hive问题

非常感谢大佬，slot我设置成了4，按照你的方法我排查了下我的问题，应该是我在我的集群配置文件sql-client-defaults.yaml文件中设置的原始值不对： 我多加了个“hive-version: 2.1.1”，后来把这一行注释掉，可以了；而且按照你的方式注册临时catalog也可以了。 这个问题导致我一直卡在这。。。哭 总之，感谢帮助哈 sql-client-defaults.yaml文件中设置的错误原始值如下： catalogs: # [] # empty list # A typical catalog definition looks like: -

How to convert local to iso 8601 time in flink sql?

Hi community, Now I have a timestamp field with format '-MM-dd HH:mm:ss', such as '2021-06-29 09:00:00'. How to convert this field to iso 8601 time with offset , such as '2021-06-29T09:00:00+08:00'? Thanks. 1095193...@qq.com

local运行模式下不会生成checkpoint吗？

我在本地使用local运行模式运行flink sql，将数据从kafka写到mongodb，mongodb connector是自己开发的，实现了CheckpointedFunction接口，debug的时候发现数据进来的时候有调用invoke方法，但没有调用initialState和snapshotState方法，我有设置enableCheckpoint，同样的程序使用kubernetes部署发现是会调用snapshotState方法。我的问题是：local运行模式下不会生成checkpoint吗？

Flink Sql 1.13 UDF ERROR

Hi, All. 请教一个在最新的1.13.1 api升级调研中遇到的问题，谢谢大家： 版本: 1.13.1 运行模式： IDE-application --- about udf define... public static class UDFAggregateFunction extends AggregateFunction { //返回最终结果 @Override public

State Processor API and existing state

Let's say that a job has operators with UIDs: W, X, Y, and Z, and uses RocksDB as a backend with checkpoint data URI s3://checkpoints" Then I stop the job with a savepoint at s3://savepoint-1. I assume that all the data within the checkpoint are stored within the given Savepoint. Is that

Re: Looking for example code

Thanks, I'll check them out. Thomas J. Raef Founder, WeWatchYourWebsite.com http://wewatchyourwebsite.com tr...@wewatchyourwebsite.com LinkedIn Facebook On Mon, Jun 28, 2021 at 11:16 AM Piotr

Re: Savepoint failure with operation not found under key

Ordinarily this happens because the status request is sent to a different JM than the one who received the request for creating a savepoint. The meta information for such requests is only stored locally on each JM and neither distributed to all JMs nor persisted anywhere. Did you send both

Re: Yarn Application Crashed?

Hi, You should still be able to get the Flink logs via: > yarn logs -applicationId application_1623861596410_0010 And it should give you more answers about what has happened. About the Flink and YARN behaviour, have you seen the documentation? [1] Especially this part: > Failed containers

Re: Looking for example code

Have you seen the documents that I linked? Isn't it enough? First pular link that I posted [4] has some example code. Literally the first link inside the second pulsar blog I referenced [5] leads to the pulsar connector repository which also has some examples [6]. Piotrek [6]

Re: Looking for example code

I need it to connect to Pulsar and stream from Pulsar. I could not find any code on how to connect to Pulsar. I've done the WordCount, but I need sample code for how to connect to Pulsar. Thomas J. Raef Founder, WeWatchYourWebsite.com http://wewatchyourwebsite.com tr...@wewatchyourwebsite.com

Re: Cancel job error ! Interrupted while waiting for buffer

Hi, It's hard to say from the log fragment, but I presume this task has correctly switched to "CANCELLED" state and this error should not have been logged as an ERROR, right? How did you get this stack trace? Maybe it was logged as a DEBUG message? If not, that would be probably a minor bug in

Re: Looking for example code

Hi, We are glad that you want to try out Flink, but if you would like to get help you need to be a bit more specific. What are you exactly doing, and what, on which step exactly and how is not working (including logs and/or error messages) is necessary for someone to help you. In terms of how to

Flink Sql 1.13 UDF ERROR

Hi, All. 请教一个在最新的1.13.1 api升级调研中遇到的问题，谢谢大家： 版本: 1.13.1 运行模式： IDE-application --- about udf define... public static class UDFAggregateFunction extends AggregateFunction { //返回最终结果 @Override public

退订

退订

退订

退订

Re: FW: Hadoop3 with Flink

Sorry for the belated reply. In 1.12, you just need to make sure that the HADOOP_CLASSPATH environment variable is set up. For more details, please refer to [1]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/ Best, Yangze Guo On Mon, Jun 28,

Re: Event Time Timers in Process functions when time characteristic is ProcessingTime

Hi lenduha, > Processing time: We can only use processing time windows & timer. Event time: We are able to use both processing time & event time windows & timers. Processing time is always wall/clock time depending on the host running task manager so independent from the event time? Yes. > Also

FW: Hadoop3 with Flink

Hi, Can anyone please share inputs on this? Regards, Suchithra From: V N, Suchithra (Nokia - IN/Bangalore) Sent: Thursday, June 24, 2021 2:35 PM To: user@flink.apache.org Subject: Hadoop3 with Flink Hello, We are using Apache flink 1.12.3 and planning to use Hadoop 3 version. Could you please

Re: Event Time Timers in Process functions when time characteristic is ProcessingTime

Thanks for the info Jing, just to clarify one point, so when we set the environment time characteristics to Processing time: We can only use processing time windows & timer. Event time: We are able to use both processing time & event time windows & timers. Processing time is always wall/clock

Re: Exception in snapshotState suppresses subsequent checkpoints

My job is very simple as you can see from the code I pasted. I simply print out the number to stdout. If you look at the log the number continued to print out after checkpoint 1 which indicated no back pressure was happening. It is very easy to reproduce this if you run the code I provided in IDE

Re: Event Time Timers in Process functions when time characteristic is ProcessingTime

Hi lenduha, > When set the time characteristics to ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see watermarks in the Flink UI. The watermark would be swallowed in the case. > can I use Event Time Timers even if I set the time characteristics to ProcessingTime (via

Re: flink sql报错: Could not find any format factory for identifier 'parquet' in the classpath

Hi, 貌似是jar包冲突了，我再确认确认。 在 2021/6/28 下午2:33，“王刚” 写入: 注意：此封邮件来自于公司外部，请注意信息安全！ Attention: This email comes from outside of the company, please pay attention to the information security! 把flink parquet包放在flink客户端lib包下试试呢

在本地环境IDEA远程调试Flink报错

目的：想在本地环境IDEA远程调试Flink 步骤： 1.这是Debug的配置项 2.报错堆栈信息： org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'Streaming WordCount'. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:374) at

Re: [Flink SQL] Lookup join hbase problem

Hi houyin, Jark, Sorry I missed Jark's response before. > We should support lookup HBase on multiple fields (by Get#setFilter). Feel free to open issues. I agree with Jark. It's better to support multiple keys for HBase. > How can I solve this problem ? Before Hbase supports multiple keys, we

Re: [Flink SQL] Lookup join hbase problem

Hi houyin, > It maybe because the conditions in where clause, being pushed down as a > predicate into join clause ? Yes, after pushdown, hbase lookupkeys are `rowKey` and `city_code`, which trigger above exception. > How can I solve this problem ? Because only constant value and field input ref

Event Time Timers in Process functions when time characteristic is ProcessingTime

Hi, In environment configuration when set the time characteristics to ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see watermarks in the Flink UI. I think this is expected, because watermarks disabled in the source (using Kafka as source)? Another point here is, can I use

Re: Exception in snapshotState suppresses subsequent checkpoints

Hi Tao, I'm afraid that your Flink job continues to be in high backpressued and all subsequent checkpoints did not ever run 'FromElementsFunctionT#snapshotState' which means your code to throw exception never be executed. You could check those expired checkpoints to see whether your tasks

Re: flink sql报错: Could not find any format factory for identifier 'parquet' in the classpath

把flink parquet包放在flink客户端lib包下试试呢 原始邮件 发件人: Wei JI10 季伟 收件人: user-zh@flink.apache.org 发送时间: 2021年6月28日(周一) 14:14 主题: Re: flink sql报错: Could not find any format factory for identifier 'parquet' in the classpath 您好， 我没有设置scope，我看jar包中是有org/apache/flink/formats/parquet/这个目录的... 在

Re: [Flink SQL] Lookup join hbase problem

Yes. Currently, the HBase lookup source only supports lookup on rowkey. If there is more than one join on condition, it may fail. We should support lookup HBase on multiple fields (by Get#setFilter). Feel free to open issues. Best, Jark On Mon, 28 Jun 2021 at 12:48, 纳兰清风 wrote: > Hi, > >

Re: Questions about UPDATE_BEFORE row kind in ElasticSearch sink

UPDATE_BEFORE is required in cases such as Aggregation with Filter. For example: SELECT * FROM ( SELECT word, count(*) as cnt FROM T GROUP BY word ) WHERE cnt < 3; There is more discussion in this issue: https://issues.apache.org/jira/browse/FLINK-9528 Best, Jark On Mon, 28 Jun 2021 at

Re: flink sql报错: Could not find any format factory for identifier 'parquet' in the classpath

您好， 我没有设置scope，我看jar包中是有org/apache/flink/formats/parquet/这个目录的... 在 2021/6/28 下午12:47，“zhisheng” 写入: 注意：此封邮件来自于公司外部，请注意信息安全！ Attention: This email comes from outside of the company, please pay attention to the information security! 看下你引入的 jar 包是咋引入的，scope 设置的是 provided 吧？