Re: Flink state migration from 1.9 to 1.13

2022-04-15 Thread XU Qinghui
> https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#table-api--sql > [2] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336489 > > On Thu, 14 Apr 2022 a

Re: Flink state migration from 1.9 to 1.13

2022-04-14 Thread XU Qinghui
> lambda (keyBy(e->e.getId()) in 1.13. This will make the key serializer > incompatible. > > You may reference this link for how to use Apache Flink’s State Processor > API to modifying savepoints: > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/libs/state_pro

Flink state migration from 1.9 to 1.13

2022-04-13 Thread XU Qinghui
Hello dear community, We are trying to upgrade our flink from 1.9 to 1.13, but it seems the same job running in 1.13 cannot restore the checkpoint / savepoint created by 1.9. The stacktrace looks like: java.lang.Exception: Exception while creating StreamOperatorStateContext. at org.apache.fli

Re: State Schema Evolution within SQL API

2021-03-04 Thread XU Qinghui
Hello Dawid I'm interested in this discussion because I'm currently trying to upgrade flink from 1.9 to 1.12 for a bunch of sql jobs running in production. From what you said, this seems to be a risky operation which might corrupt the states. Is there any recommendation to mitigate the risk? It wo

Re: Need help with JDBC Broken Pipeline Issue after some idle time

2021-03-02 Thread XU Qinghui
It sounds like the jdbc driver's connection is closed somehow, and probably has nothing to do with flink itself. Maybe you could check if there's some settings on the db that could close the connection after some inactivity, or otherwise it could be your network drops the inactive tcp connection af

Re: Savepoint documentation

2021-03-02 Thread XU Qinghui
Out of curiosity, does it mean that savepoint created by flink 1.11 cannot be recovered by a job running with flink 1.10 or older versions (so downgrade is impossible)? Le mar. 2 mars 2021 à 12:25, David Anderson a écrit : > You are correct in thinking that the documentation wasn't updated. If y

How to get operator uid from a sql

2021-03-02 Thread XU Qinghui
Hello folks I'm trying to use the flink state processor api to read the state of operators from a checkpoint. But currently the operator look up in the API relies on the operator `uid` (e.g. ExistingSavepoint.readKeyedState(uid, readerFunction)). But when it comes to a sql job, where should I look

Re: flink configuration: best practice for checkpoint storage secrets

2020-10-08 Thread XU Qinghui
le/ops/filesystems/s3.html#configure-access-credentials > > Cheers, > Till > > On Wed, Oct 7, 2020 at 12:31 PM XU Qinghui wrote: > >> Hello, folks >> >> We are trying to use S3 for the checkpoint storage, and this >> involves some secrets in the configuration. We

flink configuration: best practice for checkpoint storage secrets

2020-10-07 Thread XU Qinghui
Hello, folks We are trying to use S3 for the checkpoint storage, and this involves some secrets in the configuration. We tried two approaches to configure those secrets: - in the jvm application argument for jobmanager and taskmanager, such as -Ds3.secret-key - in the flink-conf.yaml file for jobm