退订

2021-08-27 文章 15678617442 
退订


| |
罗海芳
|
|
邮箱：15678617...@163.com
|

签名由 网易邮箱大师 定制

Re: 如何 Flink 通过不同的hive_catalog，访问离线和实时两套集群的hive？

2021-08-27 文章 Jim Chen 
Hi, Caizhi

你们有实时集群和离线集群分离的场景吗？你们是怎么做的？


Caizhi Weng  于2021年8月27日周五 下午2:33写道：

> Hi！
>
> 看起来是 Flink 集群不能访问到 wx12-dsjrt-master001/xx.xx.xx.xx:8050，检查一下网络以及这个端口的状态看看？
>
> Jim Chen  于2021年8月27日周五 下午1:59写道：
>
> > Hi, All
> >
> >
> 我是使用的flink1.13.1版本，我们有2套Hadoop集群，离线和实时集群。现在实时集群上的任务，想通过hive_catalog访问离线集群。
> >   按照官网例子，我分别配置离线和实时的hive-conf-dir的不同路径，发现一只报错。如：
> > 2021-08-27 13:50:22,902 INFO  org.apache.hadoop.ipc.Client
> > [] - Retrying connect to server:
> > wx12-dsjrt-master001/xx.xx.xx.xx:8050. Already tried 6 time(s); retry
> > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50,
> sleepTime=1000
> > MILLISECONDS)。
> >   大家在生产中是如何解决这种问题的呢？非常感谢！
> >
>

Re: 如何 Flink 通过不同的hive_catalog，访问离线和实时两套集群的hive？

2021-08-27 文章 Jim Chen 
Hi

集群上根本没有这个端口，也不知道这个端口是干嘛用的，为啥要连这个。这个是实时集群上的端口

Caizhi Weng  于2021年8月27日周五 下午2:33写道：

> Hi！
>
> 看起来是 Flink 集群不能访问到 wx12-dsjrt-master001/xx.xx.xx.xx:8050，检查一下网络以及这个端口的状态看看？
>
> Jim Chen  于2021年8月27日周五 下午1:59写道：
>
> > Hi, All
> >
> >
> 我是使用的flink1.13.1版本，我们有2套Hadoop集群，离线和实时集群。现在实时集群上的任务，想通过hive_catalog访问离线集群。
> >   按照官网例子，我分别配置离线和实时的hive-conf-dir的不同路径，发现一只报错。如：
> > 2021-08-27 13:50:22,902 INFO  org.apache.hadoop.ipc.Client
> > [] - Retrying connect to server:
> > wx12-dsjrt-master001/xx.xx.xx.xx:8050. Already tried 6 time(s); retry
> > policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50,
> sleepTime=1000
> > MILLISECONDS)。
> >   大家在生产中是如何解决这种问题的呢？非常感谢！
> >
>

flink oss ha

2021-08-27 文章 dker eandei 
您好:

看文档OSS可以用作 FsStatebackend，那么Flink on k8s 
做高可用时，high-availability.storageDir可以配置成oss吗，我试了下，报以下错误:

[cid:image002.png@01D79B53.F4C71E80]

从 Windows 版邮件发送

Re: flink sink oss problem

2021-08-27 文章 payne_z 
你可以看一下我上面贴的那个PR，那个PR实现的功能就是datastream sink to oss。至于写法就是和Datastream sink to 
hdfs类似的。
return StreamingFileSink
 .forRowFormat(new Path(ossPath), new SimpleStringEncoder("UTF-8"))
 .build();




在 2021年8月27日 14:34，Caizhi Weng 写道：


Hi！ 从报错上来看好像是类冲突的问题。不过 oss sink 目前应该也不支持流式写入。 具体是怎么把 datastream 写入 oss 
呢？方便的话可以把代码贴一下，帮助大家更好观察问题。 payne_z  于2021年8月27日周五 
下午2:02写道： > 当我使用 https://github.com/apache/flink/pull/7798， 这个PR打pacth到flink > 
1.11.2，发现写flink通过Datastream写OSS功能并不可用。 > 报错信息如下： > java.lang.RuntimeException: 
java.lang.RuntimeException: class > 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2624)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:106)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:101)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:448)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:331)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:298)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1780)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:704)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:654)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:565)
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:336)
 > at > 
org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:105)
 > at > 
org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:61)
 > at > 
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:468) 
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > at > 
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260)
 > at > 
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396)
 > at > 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
 > at > 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
 > at > 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
 > at > 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
 > at > 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
 > at > 
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
 > at > 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
 > at > 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
 > at > 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
 > at > 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) 
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > at 
java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: 
class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider
 > at > 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2618)
 > ... 29 more

Re: flink sink oss problem

2021-08-27 文章 Caizhi Weng 
Hi！

从报错上来看好像是类冲突的问题。不过 oss sink 目前应该也不支持流式写入。

具体是怎么把 datastream 写入 oss 呢？方便的话可以把代码贴一下，帮助大家更好观察问题。

payne_z  于2021年8月27日周五 下午2:02写道：

> 当我使用 https://github.com/apache/flink/pull/7798， 这个PR打pacth到flink
> 1.11.2，发现写flink通过Datastream写OSS功能并不可用。
> 报错信息如下：
> java.lang.RuntimeException: java.lang.RuntimeException: class
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2624)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:106)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:101)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:448)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:331)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:298)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1780)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:704)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:654)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:565)
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:336)
> at
> org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:105)
> at
> org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:61)
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:468)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260)
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: class
> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2618)
> ... 29 more

Re: 如何 Flink 通过不同的hive_catalog，访问离线和实时两套集群的hive？

2021-08-27 文章 Caizhi Weng 
Hi！

看起来是 Flink 集群不能访问到 wx12-dsjrt-master001/xx.xx.xx.xx:8050，检查一下网络以及这个端口的状态看看？

Jim Chen  于2021年8月27日周五 下午1:59写道：

> Hi, All
>
> 我是使用的flink1.13.1版本，我们有2套Hadoop集群，离线和实时集群。现在实时集群上的任务，想通过hive_catalog访问离线集群。
>   按照官网例子，我分别配置离线和实时的hive-conf-dir的不同路径，发现一只报错。如：
> 2021-08-27 13:50:22,902 INFO  org.apache.hadoop.ipc.Client
> [] - Retrying connect to server:
> wx12-dsjrt-master001/xx.xx.xx.xx:8050. Already tried 6 time(s); retry
> policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000
> MILLISECONDS)。
>   大家在生产中是如何解决这种问题的呢？非常感谢！
>

flink sink oss problem

2021-08-27 文章 payne_z 
当我使用 https://github.com/apache/flink/pull/7798， 这个PR打pacth到flink 
1.11.2，发现写flink通过Datastream写OSS功能并不可用。
报错信息如下：
java.lang.RuntimeException: java.lang.RuntimeException: class 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2624)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:106)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:101)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:448)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:331)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:298)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1780)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:704)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:654)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:565)
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:336)
at 
org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:105)
at 
org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:61)
at 
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:468)
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389)
at 
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260)
at 
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396)
at 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185)
at 
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167)
at 
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at 
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: class 
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider
at 
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2618)
... 29 more