回复：退订
Hi 退订应该发送到 user-zh-unsubscr...@flink.apache.org Best JasonLee 在2021年08月21日 09:43，牛成 写道： 退订
退订
退订
退订
退订
退订
Re:Re:Re:flink checkpoint 到hdfs 报错
集群里的机器互相访问配的是内网地址呗，你这得开内网访问...
在 2021-08-20 18:56:58，"杨帅统" 写道：
>
>
>
>
>
>
>test.gl.cdh.node1 对应的是远程服务器外网地址 139.9.132.*
>192.168.0.32:9866是139.9.132.*机器的同一内网下的另一台内网地址 为啥会返回内网地址啊。。。
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>在 2021-08-20 18:28:34，"东东" 写道：
>>这不很清楚么，连 192.168.0.32:9866 超时啊
>>
>>
>>
>>
>>在 2021-08-20 18:13:10，"杨帅统" 写道：
>>>// 开启checkpoint
>>>env.enableCheckpointing(5000L, CheckpointingMode.EXACTLY_ONCE);
>>>
>>>env.getCheckpointConfig().setCheckpointStorage("hdfs://test.gl.cdh.node1:8020/flink/flink-cdc-demo");
>>>System.setProperty("HADOOP_USER_NAME", "root");
>>>
>>>
>>>报错信息如下：
>>>org.apache.hadoop.net.ConnectTimeoutException: 6 millis timeout while
>>>waiting for channel to be ready for connect. ch :
>>>java.nio.channels.SocketChannel[connection-pending remote=/192.168.0.32:9866]
>>>at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
>>>at
>>>org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
>>>at
>>>org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
>>>at
>>>org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
>>>at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
>>>2021-08-20 18:11:49 WARN DataStreamer:1683 - Abandoning
>>>BP-1233461189-192.168.0.27-1615278916172:blk_1074013541_272740
>>>2021-08-20 18:11:50 WARN DataStreamer:1688 - Excluding datanode
>>>DatanodeInfoWithStorage[192.168.0.32:9866,DS-f45126b3-f020-473f-b25f-1b37f8540eb7,DISK]
>>>2021-08-20 18:11:50 WARN DataStreamer:826 - DataStreamer Exception
>>>org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
>>>/flink/flink-cdc-demo/13714486ceb74d650ba104df7b202920/chk-1/_metadata could
>>>only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s)
>>>running and 3 node(s) are excluded in this operation.
>>>at
>>>org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2102)
>>>at
>>>org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
>>>at
>>>org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2673)
Re:Re:flink checkpoint 到hdfs 报错
test.gl.cdh.node1 对应的是远程服务器外网地址 139.9.132.*
192.168.0.32:9866是139.9.132.*机器的同一内网下的另一台内网地址 为啥会返回内网地址啊。。。
在 2021-08-20 18:28:34，"东东" 写道：
>这不很清楚么，连 192.168.0.32:9866 超时啊
>
>
>
>
>在 2021-08-20 18:13:10，"杨帅统" 写道：
>>// 开启checkpoint
>>env.enableCheckpointing(5000L, CheckpointingMode.EXACTLY_ONCE);
>>
>>env.getCheckpointConfig().setCheckpointStorage("hdfs://test.gl.cdh.node1:8020/flink/flink-cdc-demo");
>>System.setProperty("HADOOP_USER_NAME", "root");
>>
>>
>>报错信息如下：
>>org.apache.hadoop.net.ConnectTimeoutException: 6 millis timeout while
>>waiting for channel to be ready for connect. ch :
>>java.nio.channels.SocketChannel[connection-pending remote=/192.168.0.32:9866]
>>at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
>>at
>>org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
>>at
>>org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
>>at
>>org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
>>at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
>>2021-08-20 18:11:49 WARN DataStreamer:1683 - Abandoning
>>BP-1233461189-192.168.0.27-1615278916172:blk_1074013541_272740
>>2021-08-20 18:11:50 WARN DataStreamer:1688 - Excluding datanode
>>DatanodeInfoWithStorage[192.168.0.32:9866,DS-f45126b3-f020-473f-b25f-1b37f8540eb7,DISK]
>>2021-08-20 18:11:50 WARN DataStreamer:826 - DataStreamer Exception
>>org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
>>/flink/flink-cdc-demo/13714486ceb74d650ba104df7b202920/chk-1/_metadata could
>>only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s)
>>running and 3 node(s) are excluded in this operation.
>>at
>>org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2102)
>>at
>>org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
>>at
>>org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2673)
Re:flink checkpoint 到hdfs 报错
这不很清楚么，连 192.168.0.32:9866 超时啊
在 2021-08-20 18:13:10，"杨帅统" 写道：
>// 开启checkpoint
>env.enableCheckpointing(5000L, CheckpointingMode.EXACTLY_ONCE);
>
>env.getCheckpointConfig().setCheckpointStorage("hdfs://test.gl.cdh.node1:8020/flink/flink-cdc-demo");
>System.setProperty("HADOOP_USER_NAME", "root");
>
>
>报错信息如下：
>org.apache.hadoop.net.ConnectTimeoutException: 6 millis timeout while
>waiting for channel to be ready for connect. ch :
>java.nio.channels.SocketChannel[connection-pending remote=/192.168.0.32:9866]
>at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
>at
>org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
>at
>org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
>at
>org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
>at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
>2021-08-20 18:11:49 WARN DataStreamer:1683 - Abandoning
>BP-1233461189-192.168.0.27-1615278916172:blk_1074013541_272740
>2021-08-20 18:11:50 WARN DataStreamer:1688 - Excluding datanode
>DatanodeInfoWithStorage[192.168.0.32:9866,DS-f45126b3-f020-473f-b25f-1b37f8540eb7,DISK]
>2021-08-20 18:11:50 WARN DataStreamer:826 - DataStreamer Exception
>org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
>/flink/flink-cdc-demo/13714486ceb74d650ba104df7b202920/chk-1/_metadata could
>only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s)
>running and 3 node(s) are excluded in this operation.
>at
>org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2102)
>at
>org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
>at
>org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2673)
flink checkpoint 到hdfs 报错
// 开启checkpoint
env.enableCheckpointing(5000L, CheckpointingMode.EXACTLY_ONCE);
env.getCheckpointConfig().setCheckpointStorage("hdfs://test.gl.cdh.node1:8020/flink/flink-cdc-demo");
System.setProperty("HADOOP_USER_NAME", "root");
报错信息如下：
org.apache.hadoop.net.ConnectTimeoutException: 6 millis timeout while
waiting for channel to be ready for connect. ch :
java.nio.channels.SocketChannel[connection-pending remote=/192.168.0.32:9866]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534)
at
org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:253)
at
org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1725)
at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
2021-08-20 18:11:49 WARN DataStreamer:1683 - Abandoning
BP-1233461189-192.168.0.27-1615278916172:blk_1074013541_272740
2021-08-20 18:11:50 WARN DataStreamer:1688 - Excluding datanode
DatanodeInfoWithStorage[192.168.0.32:9866,DS-f45126b3-f020-473f-b25f-1b37f8540eb7,DISK]
2021-08-20 18:11:50 WARN DataStreamer:826 - DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/flink/flink-cdc-demo/13714486ceb74d650ba104df7b202920/chk-1/_metadata could
only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s)
running and 3 node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2102)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2673)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:872)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:550)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1499)
at org.apache.hadoop.ipc.Client.call(Client.java:1445)
at org.apache.hadoop.ipc.Client.call(Client.java:1355)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy184.addBlock(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:497)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy185.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.addBlock(DFSOutputStream.java:1085)
at
org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1865)
at
org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1668)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
2021-08-20 18:11:50 WARN JobMaster:121 - Error while processing
AcknowledgeCheckpoint message
org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize the
pending checkpoint 1. Failure reason: Failure to finalize checkpoint.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1199)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1072)
at
Re:Re: cumulate累加函数输出问题
你好： 我使用的场景是要实时统计一天的数据，在小窗口进行即时输出，延迟不能太高，cumulate window符合要求，tumble window 延迟太高了。 在 2021-08-20 16:01:57，"Caizhi Weng" 写道： >Hi！ > >你可能想要的是 tumble window 而不是 cumulate window。 > >李航飞 于2021年8月20日周五 下午3:26写道： > >> 能不能通过设置，每分钟只输出 当前小窗口中计算数据的结果，之前计算的结果不再输出 >> >> >> 目前测试发现，输入的数据越多，到下次输出的数据也会越来越多， >> 不同窗口的计算结果，都会再下次窗口中输出， >> >>
Re: cumulate累加函数输出问题
Hi！ 你可能想要的是 tumble window 而不是 cumulate window。 李航飞 于2021年8月20日周五 下午3:26写道： > 能不能通过设置，每分钟只输出 当前小窗口中计算数据的结果，之前计算的结果不再输出 > > > 目前测试发现，输入的数据越多，到下次输出的数据也会越来越多， > 不同窗口的计算结果，都会再下次窗口中输出， > >
Re: 怎么才能取消订阅呢
To remove your address from the list, send a message to: mailto:user-zh-unsubscr...@flink.apache.org>> > 2021年8月20日 下午3:36，18221112048 <18221112...@163.com> 写道： > >
怎么才能取消订阅呢
spring????flinkclient????jar??????????
?? spring??jarjarflink??flinkclientjar??spring??jar??webjar??flinkjar??web?? java.util.zip.ZipException: invalid distance too far back at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164) at org.springframework.boot.loader.jar.ZipInflaterInputStream.read(ZipInflaterInputStream.java:52) at java.io.FilterInputStream.read(FilterInputStream.java:107) at java.util.Properties$LineReader.readLine(Properties.java:435) at java.util.Properties.load0(Properties.java:353) at java.util.Properties.load(Properties.java:341) at org.apache.flink.api.java.utils.ParameterTool.fromPropertiesFile(ParameterTool.java:137) at com.chinaunicom.streamjob.job.HawkeyeTask.main(HawkeyeTask.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219) at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:158) at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:82) at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:117) at com.chinaunicom.dataengine.graph.JobManager.submitJarJob(JobManager.java:422) at org.springblade.modules.dstream.service.impl.JobServiceImpl.submit(JobServiceImpl.java:295) java.util.zip.ZipException: invalid block type at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164) at org.springframework.boot.loader.jar.ZipInflaterInputStream.read(ZipInflaterInputStream.java:52) at java.io.FilterInputStream.read(FilterInputStream.java:107) at java.util.Properties$LineReader.readLine(Properties.java:435) at java.util.Properties.load0(Properties.java:353) at java.util.Properties.load(Properties.java:341) at org.apache.flink.api.java.utils.ParameterTool.fromPropertiesFile(ParameterTool.java:137) at com.chinaunicom.streamjob.job.CbssCdrCheatTask.main(CbssCdrCheatTask.java:54) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219) org.apache.flink.client.program.ProgramInvocationException: The program plan could not be fetched - the program aborted pre-maturely. System.out: (none)
cumulate累加函数输出问题
能不能通过设置，每分钟只输出 当前小窗口中计算数据的结果，之前计算的结果不再输出 目前测试发现，输入的数据越多，到下次输出的数据也会越来越多， 不同窗口的计算结果，都会再下次窗口中输出，
Re: 请问同一个flink history server能够支持多个flink application cluster吗？
Hi Chenyu, 這確實是目前尚未解決的一個問題，相關的 jira issue 可以看這 [1]。 jira issue 底下的討論串有提到一個替代方案是：使用 -D\$internal.pipeline.job-id=$(cat /proc/sys/kernel/random/uuid|tr -d "-") 主動為 application 模式的任務產生隨機的 jobid。 但由於此配置參數屬於 flink 內部參數，可能不保證未來任何改動後的向後兼容性，請謹慎考慮後再使用。 [1] https://issues.apache.org/jira/browse/FLINK-19358 Chenyu Zheng 於 2021年8月20日 週五 下午12:16寫道： > History Server的API也是使用jobid作为区分 > > * /config > * /jobs/overview > * /jobs/ > * /jobs//vertices > * /jobs//config > * /jobs//exceptions > * /jobs//accumulators > * /jobs//vertices/ > * /jobs//vertices//subtasktimes > * /jobs//vertices//taskmanagers > * /jobs//vertices//accumulators > * /jobs//vertices//subtasks/accumulators > * /jobs//vertices//subtasks/ > * > /jobs//vertices//subtasks//attempts/ > * > > /jobs//vertices//subtasks//attempts//accumulators > * /jobs//plan > > > From: Chenyu Zheng > Reply-To: "user-zh@flink.apache.org" > Date: Friday, August 20, 2021 at 11:43 AM > To: "user-zh@flink.apache.org" > Subject: 请问同一个flink history server能够支持多个flink application cluster吗？ > > 您好， > > 我们目前在k8s上以flink application模式运行作业，现在希望部署一个history > server方便debug。但是根据文档，flink > historyserver貌似只支持单个cluster下不同job的使用方法，如果存在多个cluster，相同的jobID将会出现错误。 > > 请问对于多个application cluster，history使用的最佳姿势是什么样的？ > > 谢谢[cid:image001.png@01D795B8.6430A670] >
