Re: Setting different idleStateRetentionTime for different queries executed in the same TableEnvironment in Flink 1.10

Thank you for answering! I was reading StreamExecutionEnvironmentImpl/StreamPlanner, and it seems to me that when trying to convert tables to DataStreams, planner.translate is taking the current tableConfig into account (aa in it reads the current tableConfig content even though it’s not explic

Re: Setting different idleStateRetentionTime for different queries executed in the same TableEnvironment in Flink 1.10

Hi Jiahui, QueryConfig is deprecated and will be removed in the future, because it is confusing that TableAPI has so many different config classes. If you want to set different idleStateRetentionTime for different queries, you can set a new idleStateRetentionTime on TableConfig before execute/subm

Re: Temporal Table Function in SQL

Hi, - Temporal Table It is implemented as dimension table, you can define a dimension table using DDL, for example, JDBC [1]. - Temporal Table Function Currently it cannot created from pure SQL, you can create it through table api[2], and then use it in sql. [1] https://ci.apache.org/projects/fl

Re: Storing Operator state in RocksDb during runtime - plans

Hi KristoffSC I'm not aware of any concrete plans for such a feature. I also CCed Yu, he may give more information about this. Best, Congxian Fabian Hueske 于2020年4月7日周二 上午4:27写道： > Hi Kristoff, > > I'm not aware of any concrete plans for such a feature. > > Best, > Fabian > > Am So., 5. Apr.

Re: Making job fail on Checkpoint Expired?

Hi Robin Glad to hear that my reply can help. >From my side, I do not think concurrent checkpoints can help, because it may cause more disk pressure problems. Currently, this is an issue[1] wants to support Unalign checkpoint, unaligned checkpoint wants to fix the problem of checkpoint under bac

Re: FlinkRuntimeException: Unexpected list element deserialization failure

Hi As Yun said, could you please share the whole stack trace(normally, you can copy more lines below the given stack trace), can not tell why this happened from the given stack trace. Best, Congxian Yun Tang 于2020年4月10日周五 上午1:48写道： > Hi > > I think you have missed the "caused by" exception [1

Re: Setting different idleStateRetentionTime for different queries executed in the same TableEnvironment in Flink 1.10

Just looked into the source code a bit further and realized that for StreamTableEnvironmentImpl, even for sinks it's also doing translation lazily. Any way we can have different transformation to have different queryConfig? From: Jiahui Jiang Sent: Friday, April

Setting different idleStateRetentionTime for different queries executed in the same TableEnvironment in Flink 1.10

Hello! I'm using Table API to write a pipeline with multiple queries. And I want to set up different idleStateRetentionTime for different queries. In Flink 1.8, it seems to be the case where I can pass in a streamQueryConfig when converting each output table into datastreams. And the translate w

Re: [PROPOSAL] Contribute training materials to Apache Flink

Yes, the training materials could become part of the docs, as you suggest. The reason why I'm somewhat resistant to this idea is that the documentation is versioned. It will be easier to maintain the training content and its exercises and examples if we only have to be concerned with having one ver

Re: upgrade flink from 1.9.1 to 1.10.0 on EMR

Hi Robert, attached the full application log file. Thanks, Anuj Container: container_1585301409143_0044_01_01 on ip-172-25-2-209.ap-south-1.compute.internal_8041 =

Flink job consuming all available memory on host

We are having an issue with a Flink Job that gradually consumes all available memory on a Docker host machine, crashing the machine. * We are running Flink 1.10.0 * We are running Flink in a Docker container on AWS ECS with EC2 instances * The Flink task manager UI does not show high memory usage

Temporal Table Function in SQL

Hi Is it possible to create a Temporal Table and a Temporal Table Function from the SQL API? Any example of DDL? Thanks

Re: [PROPOSAL] Contribute training materials to Apache Flink

+1 to the idea, David regarding your question, could it go in a quickstart/get started with Flink section? Next to or replacing the current 'Getting started' section? On Thu, 9 Apr 2020, 22:11 Seth Wiesman, wrote: > Hi David, > > +1 to add to the project. > > I agree that flink.apache.org and fl

Re: how to hold a stream until another stream is drained?

+Sun Haibo who added that validation in https://issues.apache.org/jira/browse/FLINK-11879 Hi Haibo, Any suggestion how to enable checkpointing for InputSelectable and BounedMultiInput? Thanks, Yubao Liu On 2020/4/10, 10:21 PM, "刘宇宝" wrote: Hi Fabian, Thank you very much, I

Re: how to hold a stream until another stream is drained?

Hi Fabian, Thank you very much, I almost get it working with InputSelectable: DataStream binlogStream = env.addSource(new FlinkKafkaConsumer(…)); DataStream snapshotStream = env.createInput(JDBCInputFormat.buildJDBCInputFormat()….); DataStream tableStream = snapshotStream.c

Re: Is it possible to emulate keyed state with operator state?

Hypothetically, yes, I think this is possible to some extent. You would have to give up all the things that require a KeyedStream, such as timers, and the RocksDB state backend. And performance would suffer. As for the question of determining which key groups (and ultimately, which keys) are handl

Re: Inserting nullable data into NOT NULL columns

Thanks Timo for the answer! We are specifically working on implementing Kudu Table connectors and catalog and we hit a problem while handling keys. Naturally columns that will be used as keys in Kudu cannot be null so we set their type when the catalog returns it to NOT NULL. Unfortunately this m

on YARN question

Hello, everyone! I have a question for you [root@hadoop131 bin]# ./yarn-session.sh -n 2 -s 2 -jm 1024 -tm 1024 -nm test -d    The following command reports an error ./flink run -m yarn-cluster -c com.scala.DateSetWcApp  /opt/module/datas/flink/Flink_test-1.7.2.jar  --input /opt/module/da

Re: Re: fink sql client not able to read parquet format table

https://issues.apache.org/jira/browse/FLINK-17086 It is my first time to create a flink jira issue. Just point it out and correct it if I write something wrong. Thanks, Lei wangl...@geekplus.com.cn From: Jingsong Li Date: 2020-04-10 11:03 To: wangl...@geekplus.com.cn CC: Jark Wu; lirui;

Re: Possible memory leak in JobManager (Flink 1.10.0)?

What you could also try out is whether the same problem occurs with Flink 1.7.3. We did the executor change in this bug fix release. This could help us validating my suspicion. Cheers, Till On Thu, Apr 9, 2020 at 4:24 PM Till Rohrmann wrote: > For further reference, I've created this issue [1]

Re: Javadocs Broken?

s/out/our/

Javadocs Broken?

Hi guys, Right now when I click "JavaDocs" in out docsite[1] it jumps to a page[2] I think is definitely not out api documentation. Any thoughts? Best, tison. [1] https://ci.apache.org/projects/flink/flink-docs-master/ [2] https://ci.apache.org/projects/flink/flink-docs-master/api/java/

Re: Re: Multiple SQL Optimization

Hi forideal, You are using `StreamTableEnvironment` which doesn't support multi-sink optimization in 1.10 :) You should change `StreamTableEnvironment.create` to `TableEnvironment.create`. Btw, StreamTableEnvironment will also support multi-sink optimization in 1.11. Best, Jark On Fri, 10 Apr

Re:Re: Multiple SQL Optimization

Hi Jark Thanks for your reploy. In my code, i use `TableEnvironemnt` with Blink planner. this.tableEnv = StreamTableEnvironment.create( env, EnvironmentSettings.newInstance() .useBlinkPlanner() .inStreamingMode() .build()); // update three sql sqlsWithoutFunc.forEach(sql -> { tableEnv.sqlUpdate(

Re: Multiple SQL Optimization

Hi forideal, Are you using `StreamTableEnvironment` or SQL CLI? Currently, only `TableEnvironemnt` with Blink planner have the multi-sink optimization (reuse shared upstream operators). Best, Jark On Fri, 10 Apr 2020 at 16:31, forideal wrote: > Hello > >There are 3 SQLs all querying the sa

Re: Multiple SQL Optimization

Hi forideal, Currently, Blink planner with TableEnvironment supports multiple sinks optimization which will try best to reuse common sub-graph. Best, Godfrey forideal 于2020年4月10日周五 下午4:31写道： > Hello > >There are 3 SQLs all querying the same table, but the generated GAG is > 3 independent t

Multiple SQL Optimization

Hello There are 3 SQLs all querying the same table, but the generated GAG is 3 independent topologies.I think, the better result is that there is one Source and 3 Sinks. createtablegood_sink(datavarchar)with( 'connector.type'='console', 'connector.dry-run'='false','connector.property

Re: on YARN question

Hi, I think you need to remove "-n 2" from your start up command. The "-n" option has been removed and no longer take effect. Flink will stop parsing the command line options once it see an un-recognized token. That means all the options after "-n 2" are ignored, including "-d". As a result, you'r

Re: job doesn't start via cli after migrating Flink from 1.8 to 1.10

I am trying to answer your question inline. > The server has twice more than that, and on flink 1.8 this configuration > works, why when switching to 1.10 it is not enough resources? >From 1.10, the taskmanager resource related configuration has changed and the default value is bigger than befo