回复: flink oss ha

2021-09-10 文章 wang xiguang 
抱歉图片又被吞了，附件是图片

从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送

发件人: wang xiguang<mailto:wangxg...@outlook.com>
发送时间: 2021年9月10日 14:09
收件人: user-zh@flink.apache.org<mailto:user-zh@flink.apache.org>
主题: 回复: flink oss ha

您好：
   
确认endpoint,ak,sk都正确,又尝试使用oss做状态后端，也是一样的报错。[cid:image003.png@01D7A64D.72A24A10]
看报错是com.aliyun.oss.OSSClient.listObjects方法报的。于是我用普通springboot程序使用oss客户端调用该方法，结果调用正常:
[cid:image005.png@01D7A64D.72A24A10]
使用的aliyun-sdk-oss版本与flink插件flink-oss-fs-hadoop-1.13.2.jar中的一致(3.4.1)，oss的endpoint,bucket一致。
如下是flink集群配置:
[cid:image008.png@01D7A64D.72A24A10]
特别地，endpoint添加了‘http://‘，不添加会访问到endpoint的443端口。
从 Windows 
版邮件<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986=04%7C01%7C%7C65a9920c7e6141e7cf1708d974219773%7C84df9e7fe9f640afb435%7C1%7C0%7C637668509908741063%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=Kh3kUoQVKwjXBcYjB0M5%2B1cNo5Hr7xcYRj7gYESz1YE%3D=0>发送

发件人: Yun Tang<mailto:myas...@live.com>
发送时间: 2021年8月31日 11:02
收件人: user-zh@flink.apache.org<mailto:user-zh@flink.apache.org>
主题: Re: flink oss ha

Hi

这个看上去更像是oss配置的问题，你能使用目前配置的 oss.endpoint，accessKeyId以及accessKeySecret 
结合ossutil等工具访问对应的 oss://bucket-logcenter/flink-state/flink-session-recovery 么？

祝好
唐云

From: dker eandei 
Sent: Monday, August 30, 2021 12:36
To: user-zh@flink.apache.org 
Subject: 回复: flink oss ha

您好:
 附件是使用oss作高可用时的报错，以下是启动flink时的脚本:

../bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=flink-session-1 \
-Dkubernetes.container.image=test/flink:1.13.2-scala_2.12-oss \
-Dkubernetes.container.image.pull-policy=Always \
-Dkubernetes.namespace=flink-session \
-Dkubernetes.service-account=flink-session-sa \
-Dkubernetes.rest-service.exposed.type=ClusterIP \
-Dtaskmanager.numberOfTaskSlots=6 \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=2048m \
-Dkubernetes.jobmanager.cpu=1 \
-Dkubernetes.taskmanager.cpu=2 \

-Dfs.oss.endpoint="https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Foss-.local%2Fdata=04%7C01%7C%7Ca51e8e8ec8494e3276c208d96c2bb7ba%7C84df9e7fe9f640afb435%7C1%7C0%7C637659757312053633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=BIcRoWblYNc5tkXbwt4RdQuALpi7wxKga2xH0QPY8gw%3Dreserved=0<https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Foss-.local%2F=04%7C01%7C%7C65a9920c7e6141e7cf1708d974219773%7C84df9e7fe9f640afb435%7C1%7C0%7C637668509908741063%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=j4TLKOFPM%2BA57r7pFYV%2FbW71r5Xp4LYHuRRlgcC7Uwk%3D=0>"
 \
-Dfs.oss.accessKeyId="j0BAJ" \
-Dfs.oss.accessKeySecret="7mzTPiC4w" \

-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
 \

-Dhigh-availability.storageDir=oss://bucket-logcenter/flink-state/flink-session-recovery
 \

-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar
 \

-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar

-邮件原件-
发件人: Yun Tang 
发送时间: 2021年8月30日 11:36
收件人: user-zh@flink.apache.org
主题: Re: flink oss ha

Hi,
你好，图片无法加载，可以直接粘贴文字出来

祝好
唐云

From: dker eandei 
Sent: Friday, August 27, 2021 14:58
To: user-zh@flink.apache.org 
Subject: flink oss ha


您好:

看文档OSS可以用作 FsStatebackend，那么Flink on k8s 
做高可用时，high-availability.storageDir可以配置成oss吗，我试了下，报以下错误:

[cid:image002.png@01D79B53.F4C71E80]



从 Windows 
版邮件<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986data=04%7C01%7C%7Ca51e8e8ec8494e3276c208d96c2bb7ba%7C84df9e7fe9f640afb435%7C1%7C0%7C637659757312053633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=b6jb80lFbRGz%2BtBKVAdMAHUYKun1fQyt3JISlZKGUyk%3Dreserved=0<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986=04%7C01%7C%7C65a9920c7e6141e7cf1708d974219773%7C84df9e7fe9f640afb435%7C1%7C0%7C637668509908751033%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=evnUnGBA%2BIbq9EvLN4zNTal5W4964KVgBdJXzjfW%2BIo%3D=0>>发送

回复: flink oss ha

2021-09-10 文章 wang xiguang 
您好：
   
确认endpoint,ak,sk都正确,又尝试使用oss做状态后端，也是一样的报错。[cid:image003.png@01D7A64D.72A24A10]
看报错是com.aliyun.oss.OSSClient.listObjects方法报的。于是我用普通springboot程序使用oss客户端调用该方法，结果调用正常:
[cid:image005.png@01D7A64D.72A24A10]
使用的aliyun-sdk-oss版本与flink插件flink-oss-fs-hadoop-1.13.2.jar中的一致(3.4.1)，oss的endpoint,bucket一致。
如下是flink集群配置:
[cid:image008.png@01D7A64D.72A24A10]
特别地，endpoint添加了‘http://‘，不添加会访问到endpoint的443端口。
从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送

发件人: Yun Tang<mailto:myas...@live.com>
发送时间: 2021年8月31日 11:02
收件人: user-zh@flink.apache.org<mailto:user-zh@flink.apache.org>
主题: Re: flink oss ha

Hi

这个看上去更像是oss配置的问题，你能使用目前配置的 oss.endpoint，accessKeyId以及accessKeySecret 
结合ossutil等工具访问对应的 oss://bucket-logcenter/flink-state/flink-session-recovery 么？

祝好
唐云

From: dker eandei 
Sent: Monday, August 30, 2021 12:36
To: user-zh@flink.apache.org 
Subject: 回复: flink oss ha

您好:
 附件是使用oss作高可用时的报错，以下是启动flink时的脚本:

../bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=flink-session-1 \
-Dkubernetes.container.image=test/flink:1.13.2-scala_2.12-oss \
-Dkubernetes.container.image.pull-policy=Always \
-Dkubernetes.namespace=flink-session \
-Dkubernetes.service-account=flink-session-sa \
-Dkubernetes.rest-service.exposed.type=ClusterIP \
-Dtaskmanager.numberOfTaskSlots=6 \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=2048m \
-Dkubernetes.jobmanager.cpu=1 \
-Dkubernetes.taskmanager.cpu=2 \

-Dfs.oss.endpoint="https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Foss-.local%2Fdata=04%7C01%7C%7Ca51e8e8ec8494e3276c208d96c2bb7ba%7C84df9e7fe9f640afb435%7C1%7C0%7C637659757312053633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=BIcRoWblYNc5tkXbwt4RdQuALpi7wxKga2xH0QPY8gw%3Dreserved=0;
 \
-Dfs.oss.accessKeyId="j0BAJ" \
-Dfs.oss.accessKeySecret="7mzTPiC4w" \

-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
 \

-Dhigh-availability.storageDir=oss://bucket-logcenter/flink-state/flink-session-recovery
 \

-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar
 \

-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar

-邮件原件-
发件人: Yun Tang 
发送时间: 2021年8月30日 11:36
收件人: user-zh@flink.apache.org
主题: Re: flink oss ha

Hi,
你好，图片无法加载，可以直接粘贴文字出来

祝好
唐云

From: dker eandei 
Sent: Friday, August 27, 2021 14:58
To: user-zh@flink.apache.org 
Subject: flink oss ha


您好:

看文档OSS可以用作 FsStatebackend，那么Flink on k8s 
做高可用时，high-availability.storageDir可以配置成oss吗，我试了下，报以下错误:

[cid:image002.png@01D79B53.F4C71E80]



从 Windows 
版邮件<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986data=04%7C01%7C%7Ca51e8e8ec8494e3276c208d96c2bb7ba%7C84df9e7fe9f640afb435%7C1%7C0%7C637659757312053633%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=b6jb80lFbRGz%2BtBKVAdMAHUYKun1fQyt3JISlZKGUyk%3Dreserved=0>发送

Re: flink oss ha

2021-08-30 文章 Yun Tang 
Hi

这个看上去更像是oss配置的问题，你能使用目前配置的 oss.endpoint，accessKeyId以及accessKeySecret 
结合ossutil等工具访问对应的 oss://bucket-logcenter/flink-state/flink-session-recovery 么？

祝好
唐云

From: dker eandei 
Sent: Monday, August 30, 2021 12:36
To: user-zh@flink.apache.org 
Subject: 回复: flink oss ha

您好:
 附件是使用oss作高可用时的报错，以下是启动flink时的脚本:

../bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=flink-session-1 \
-Dkubernetes.container.image=test/flink:1.13.2-scala_2.12-oss \
-Dkubernetes.container.image.pull-policy=Always \
-Dkubernetes.namespace=flink-session \
-Dkubernetes.service-account=flink-session-sa \
-Dkubernetes.rest-service.exposed.type=ClusterIP \
-Dtaskmanager.numberOfTaskSlots=6 \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=2048m \
-Dkubernetes.jobmanager.cpu=1 \
-Dkubernetes.taskmanager.cpu=2 \
-Dfs.oss.endpoint="http://oss-.local; \
-Dfs.oss.accessKeyId="j0BAJ" \
-Dfs.oss.accessKeySecret="7mzTPiC4w" \

-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
 \

-Dhigh-availability.storageDir=oss://bucket-logcenter/flink-state/flink-session-recovery
 \

-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar
 \

-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar

-邮件原件-
发件人: Yun Tang 
发送时间: 2021年8月30日 11:36
收件人: user-zh@flink.apache.org
主题: Re: flink oss ha

Hi,
你好，图片无法加载，可以直接粘贴文字出来

祝好
唐云

From: dker eandei 
Sent: Friday, August 27, 2021 14:58
To: user-zh@flink.apache.org 
Subject: flink oss ha


您好:

看文档OSS可以用作 FsStatebackend，那么Flink on k8s 
做高可用时，high-availability.storageDir可以配置成oss吗，我试了下，报以下错误:

[cid:image002.png@01D79B53.F4C71E80]



从 Windows 
版邮件<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986data=04%7C01%7C%7Cd552b12a5a1f4a92aaee08d96b674cd0%7C84df9e7fe9f640afb435%7C1%7C0%7C637658913686219405%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=9Z9mxYxfkKqWfTCHYPThD3I97KFAFRMKINYExBuge80%3Dreserved=0>发送

回复: flink oss ha

2021-08-29 文章 dker eandei 
您好:
 附件是使用oss作高可用时的报错，以下是启动flink时的脚本:

../bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=flink-session-1 \
-Dkubernetes.container.image=test/flink:1.13.2-scala_2.12-oss \
-Dkubernetes.container.image.pull-policy=Always \
-Dkubernetes.namespace=flink-session \
-Dkubernetes.service-account=flink-session-sa \
-Dkubernetes.rest-service.exposed.type=ClusterIP \
-Dtaskmanager.numberOfTaskSlots=6 \
-Djobmanager.memory.process.size=1024m \
-Dtaskmanager.memory.process.size=2048m \
-Dkubernetes.jobmanager.cpu=1 \
-Dkubernetes.taskmanager.cpu=2 \
-Dfs.oss.endpoint="http://oss-.local; \
-Dfs.oss.accessKeyId="j0BAJ" \
-Dfs.oss.accessKeySecret="7mzTPiC4w" \

-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
 \

-Dhigh-availability.storageDir=oss://bucket-logcenter/flink-state/flink-session-recovery
 \

-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar
 \

-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-oss-fs-hadoop-1.13.2.jar

-邮件原件-
发件人: Yun Tang  
发送时间: 2021年8月30日 11:36
收件人: user-zh@flink.apache.org
主题: Re: flink oss ha

Hi,
你好，图片无法加载，可以直接粘贴文字出来

祝好
唐云

From: dker eandei 
Sent: Friday, August 27, 2021 14:58
To: user-zh@flink.apache.org 
Subject: flink oss ha


您好:

看文档OSS可以用作 FsStatebackend，那么Flink on k8s 
做高可用时，high-availability.storageDir可以配置成oss吗，我试了下，报以下错误:

[cid:image002.png@01D79B53.F4C71E80]



从 Windows 
版邮件<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986data=04%7C01%7C%7Cd552b12a5a1f4a92aaee08d96b674cd0%7C84df9e7fe9f640afb435%7C1%7C0%7C637658913686219405%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=9Z9mxYxfkKqWfTCHYPThD3I97KFAFRMKINYExBuge80%3Dreserved=0>发送


2021-08-30 12:21:19,298 INFO  akka.remote.Remoting  
   [] - Remoting started; listening on addresses 
:[akka.tcp://flink@ip:6123]
2021-08-30 12:21:19,547 INFO  
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils[] - Actor system 
started at akka.tcp://flink@ip:6123
2021-08-30 12:21:21,816 INFO  org.apache.flink.runtime.blob.FileSystemBlobStore 
   [] - Creating highly available BLOB storage directory at 
oss://bucket-logcenter/flink-state/flink-session-recovery/flink-session-1/blob
2021-08-30 12:21:22,136 INFO  
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss  [] - 
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 612C5CC21078CF8B58AB7521
[HostId]: null
2021-08-30 12:21:22,145 INFO  
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss  [] - 
[Server]Unable to execute HTTP request: Not Found
[ErrorCode]: NoSuchKey
[RequestId]: 612C5CC27BF4BC5C747B6452
[HostId]: null
2021-08-30 12:21:22,467 WARN  
org.apache.flink.fs.osshadoop.shaded.com.aliyun.oss  [] - 
[Server]Unable to execute HTTP request: There are invalid characters in 
parameters.
[ErrorCode]: InvalidArgument
[RequestId]: 612C5CC2EF7A8F7D9E7B4301
[HostId]: oss-.local
[ResponseError]:


  InvalidArgument
  There are invalid characters in parameters.
  612C5CC2EF7A8F7D9E7B4301
  oss-.local
  prefix
  
flink-state/flink-session-recovery/flink-session-1/blob/


2021-08-30 12:21:22,471 INFO  
org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Shutting 
KubernetesSessionClusterEntrypoint down with application status FAILED. 
Diagnostics org.apache.flink.util.FlinkException: Could not create the ha 
services from the instantiated HighAvailabilityServicesFactory 
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
at 
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
at 
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:353)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:311)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:239)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:189)
at 
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:186)
at 
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:600)
at 
org.apache.flink.kubernetes.entrypoint.Kubernet

Re: flink oss ha

2021-08-29 文章 Yun Tang 
Hi,
你好，图片无法加载，可以直接粘贴文字出来

祝好
唐云

From: dker eandei 
Sent: Friday, August 27, 2021 14:58
To: user-zh@flink.apache.org 
Subject: flink oss ha


您好:

看文档OSS可以用作 FsStatebackend，那么Flink on k8s 
做高可用时，high-availability.storageDir可以配置成oss吗，我试了下，报以下错误:

[cid:image002.png@01D79B53.F4C71E80]



从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送

flink oss ha

2021-08-27 文章 dker eandei 
您好:

看文档OSS可以用作 FsStatebackend，那么Flink on k8s 
做高可用时，high-availability.storageDir可以配置成oss吗，我试了下，报以下错误:

[cid:image002.png@01D79B53.F4C71E80]

从 Windows 版邮件发送