Re: 如何在flink中正确使用外部数据库连接
Hi, 根据我的经验，使用连接池时，至少需要及时关掉 statement/ResultSet，否则查询的结果会一直缓存，会有内存泄漏的问题。 Best， Lijie lxk7...@163.com 于2022年7月23日周六 15:34写道： > > 目前的项目中，需要使用外部数据库进行实时的look up。实时的主流数据量一天在百万级别，能用到的外部数据库有Mysql,Clickhouse > 以及Redis缓存。 > 现在是将数据实时落到Clickhouse,然后Flink实时的去look up > clickhouse。(虽然知道Clickhouse并发性能不强，但目前能用的就只有这个了，需要存储千万级别的数据） > 测试了两种方式： > > 1.使用JDBC连接池的方式去查询，Druid连接池以及C3P0连接池都用过，但是程序都是运行一段时间就会报OOM（有可能是使用方式不对）。通过dump日志排查的时候发现连接池的方式会将很多信息保留下来，所以最终没有使用这种方式。同时的话，在flink内部使用连接池的时候也没有显示的关闭连接。只在Close方法中调用了关闭。 > 2.使用DriverManager获取连接查询。这种方式目前测试下来，程序是稳定运行的，也没有报OOM。同时也没有去关闭连接。 > > 问题：1.如何正确在flink内部使用外部数据库连接，使用数据池的方式，个人理解连接的管理都是由数据池来做的，所以不需要去显示close。同时的话，个人认为实时的程序去查，这个连接就会一直占用着，也无需关闭。简言之，无论是数据池还是直连，需不需要在invoke方法中关闭连接？还是只用在close方法中关闭连接。 > 2.这种实时的look up除了缓存之外还有没有其他更好的优化手段？或者有什么其他的方案可以替代？ > > > lxk7...@163.com >
Re: flink异常
樂 Zhanghao Chen 于2022年7月25日周一 13:39写道： > 你好，可以检查下： > > 1. tm 侧是否有异常，导致 tm 退出； > 2. tm 侧是否 gc 严重导致没有及时处理心跳； > 3. jm - tm 间是否网络有异常导致心跳信息无法传达。 > > Best, > Zhanghao Chen > > From: 陈卓宇 <2572805...@qq.com.INVALID> > Sent: Friday, July 22, 2022 11:30 > To: user-zh > Subject: flink异常 > > 社区大佬您好，小弟请教一个问题： > flink版本：1.14.5 > 异常内容如下： > 2022-07-22 10:07:51 > java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id > bdp-changlog-mid-relx-middle-promotion-dev-taskmanager-1-1 timed out. > at > org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1299) > at > org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:111) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf > .UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > 我该如何解决，如何优化 >
Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.1.0 released
Yang Wang 于2022年7月25日周一 10:55写道： > Congrats! Thanks Gyula for driving this release, and thanks to all > contributors! > > > Best, > Yang > > Gyula Fóra 于2022年7月25日周一 10:44写道： > > > The Apache Flink community is very happy to announce the release of > Apache > > Flink Kubernetes Operator 1.1.0. > > > > The Flink Kubernetes Operator allows users to manage their Apache Flink > > applications and their lifecycle through native k8s tooling like kubectl. > > > > Please check out the release blog post for an overview of the release: > > > > > https://flink.apache.org/news/2022/07/25/release-kubernetes-operator-1.1.0.html > > > > The release is available for download at: > > https://flink.apache.org/downloads.html > > > > Maven artifacts for Flink Kubernetes Operator can be found at: > > > > > https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator > > > > Official Docker image for the Flink Kubernetes Operator can be found at: > > https://hub.docker.com/r/apache/flink-kubernetes-operator > > > > The full release notes are available in Jira: > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351723 > > > > We would like to thank all contributors of the Apache Flink community who > > made this release possible! > > > > Regards, > > Gyula Fora > > >
Re: flink异常
你好，可以检查下： 1. tm 侧是否有异常，导致 tm 退出； 2. tm 侧是否 gc 严重导致没有及时处理心跳； 3. jm - tm 间是否网络有异常导致心跳信息无法传达。 Best, Zhanghao Chen From: 陈卓宇 <2572805...@qq.com.INVALID> Sent: Friday, July 22, 2022 11:30 To: user-zh Subject: flink异常 社区大佬您好，小弟请教一个问题： flink版本：1.14.5 异常内容如下： 2022-07-22 10:07:51 java.util.concurrent.TimeoutException: Heartbeat of TaskManager with id bdp-changlog-mid-relx-middle-promotion-dev-taskmanager-1-1 timed out. at org.apache.flink.runtime.jobmaster.JobMaster$TaskManagerHeartbeatListener.notifyHeartbeatTimeout(JobMaster.java:1299) at org.apache.flink.runtime.heartbeat.HeartbeatMonitorImpl.run(HeartbeatMonitorImpl.java:111) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at akka.actor.ActorCell.invoke(ActorCell.scala:561) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at akka.dispatch.Mailbox.run(Mailbox.scala:225) at akka.dispatch.Mailbox.exec(Mailbox.scala:235) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 我该如何解决，如何优化
Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.1.0 released
Congrats! Thanks Gyula for driving this release, and thanks to all contributors! Best, Yang Gyula Fóra 于2022年7月25日周一 10:44写道： > The Apache Flink community is very happy to announce the release of Apache > Flink Kubernetes Operator 1.1.0. > > The Flink Kubernetes Operator allows users to manage their Apache Flink > applications and their lifecycle through native k8s tooling like kubectl. > > Please check out the release blog post for an overview of the release: > > https://flink.apache.org/news/2022/07/25/release-kubernetes-operator-1.1.0.html > > The release is available for download at: > https://flink.apache.org/downloads.html > > Maven artifacts for Flink Kubernetes Operator can be found at: > > https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator > > Official Docker image for the Flink Kubernetes Operator can be found at: > https://hub.docker.com/r/apache/flink-kubernetes-operator > > The full release notes are available in Jira: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351723 > > We would like to thank all contributors of the Apache Flink community who > made this release possible! > > Regards, > Gyula Fora >
[ANNOUNCE] Apache Flink Kubernetes Operator 1.1.0 released
The Apache Flink community is very happy to announce the release of Apache Flink Kubernetes Operator 1.1.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications and their lifecycle through native k8s tooling like kubectl. Please check out the release blog post for an overview of the release: https://flink.apache.org/news/2022/07/25/release-kubernetes-operator-1.1.0.html The release is available for download at: https://flink.apache.org/downloads.html Maven artifacts for Flink Kubernetes Operator can be found at: https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator Official Docker image for the Flink Kubernetes Operator can be found at: https://hub.docker.com/r/apache/flink-kubernetes-operator The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351723 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Gyula Fora
