Re: flink sink oss problem
你可以看一下我上面贴的那个PR,那个PR实现的功能就是datastream sink to oss。至于写法就是和Datastream sink to hdfs类似的。 return StreamingFileSink .forRowFormat(new Path(ossPath), new SimpleStringEncoder("UTF-8")) .build(); 在 2021年8月27日 14:34,Caizhi Weng 写道: Hi! 从报错上来看好像是类冲突的问题。不过 oss sink 目前应该也不支持流式写入。 具体是怎么把 datastream 写入 oss 呢?方便的话可以把代码贴一下,帮助大家更好观察问题。 payne_z 于2021年8月27日周五 下午2:02写道: > 当我使用 https://github.com/apache/flink/pull/7798, 这个PR打pacth到flink > 1.11.2,发现写flink通过Datastream写OSS功能并不可用。 > 报错信息如下: > java.lang.RuntimeException: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2624) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:106) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:101) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:448) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:331) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:298) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1780) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:704) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:654) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:565) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:336) > at > org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:105) > at > org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:61) > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:468) > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > at > org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260) > at > org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2618) > ... 29 more
Re: flink sink oss problem
Hi! 从报错上来看好像是类冲突的问题。不过 oss sink 目前应该也不支持流式写入。 具体是怎么把 datastream 写入 oss 呢?方便的话可以把代码贴一下,帮助大家更好观察问题。 payne_z 于2021年8月27日周五 下午2:02写道: > 当我使用 https://github.com/apache/flink/pull/7798, 这个PR打pacth到flink > 1.11.2,发现写flink通过Datastream写OSS功能并不可用。 > 报错信息如下: > java.lang.RuntimeException: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2624) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:106) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:101) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:448) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:331) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:298) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1780) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:704) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:654) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:565) > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:336) > at > org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:105) > at > org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:61) > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:468) > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) > at > org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260) > at > org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) > at > org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: class > org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider > at > org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2618) > ... 29 more
flink sink oss problem
当我使用 https://github.com/apache/flink/pull/7798, 这个PR打pacth到flink 1.11.2,发现写flink通过Datastream写OSS功能并不可用。 报错信息如下: java.lang.RuntimeException: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2624) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:106) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.(Groups.java:101) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:448) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:331) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:298) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1780) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:704) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:654) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:565) at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.initialize(AliyunOSSFileSystem.java:336) at org.apache.flink.fs.osshadoop.OSSFileSystemFactory.create(OSSFileSystemFactory.java:105) at org.apache.flink.core.fs.PluginFileSystemFactory.create(PluginFileSystemFactory.java:61) at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:468) at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:389) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink$RowFormatBuilder.createBuckets(StreamingFileSink.java:260) at org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.initializeState(StreamingFileSink.java:396) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:185) at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:167) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92) at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: class org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback not org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.security.GroupMappingServiceProvider at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2618) ... 29 more