[
https://issues.apache.org/jira/browse/OOZIE-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Bodo updated OOZIE-3529:
------------------------------
Attachment: OOZIE-3529.001.patch
> Oozie not supported for s3 as filesystem
> ----------------------------------------
>
> Key: OOZIE-3529
> URL: https://issues.apache.org/jira/browse/OOZIE-3529
> Project: Oozie
> Issue Type: Bug
> Components: core
> Affects Versions: 4.3.1, 5.1.0
> Reporter: Denes Bodo
> Priority: Critical
> Labels: S3
> Attachments: OOZIE-3529.001.patch, id.pig, job.properties,
> workflow.xml
>
>
> Many customer who uses s3 file system as secondary one experiences the
> following error when Oozie tries to submit the Yarn application:
> {noformat}
> 2019-04-29 13:02:53,770 WARN ForkedActionStartXCommand:523 -
> SERVER[hwnode1.puretec.purestorage.com] USER[hrt_qa] GROUP[-] TOKEN[]
> APP[demo-wf] JOB[0000001-190423141707256-oozie-oozi-W]
> ACTION[0000001-190423141707256-oozie-oozi-W@streaming-node] Error starting
> action [streaming-node]. ErrorType [ERROR], ErrorCode
> [UnsupportedOperationException], Message [UnsupportedOperationException:
> Accessing local file system is not allowed]
> org.apache.oozie.action.ActionExecutorException:
> UnsupportedOperationException: Accessing local file system is not allowed
> at
> org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1092)
> at
> org.apache.oozie.action.hadoop.MapReduceActionExecutor.createLauncherConf(MapReduceActionExecutor.java:309)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1197)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1472)
> at
> org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:234)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:41)
> at
> org.apache.oozie.command.wf.ForkedActionStartXCommand.execute(ForkedActionStartXCommand.java:30)
> at org.apache.oozie.command.XCommand.call(XCommand.java:287)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.UnsupportedOperationException: Accessing local file
> system is not allowed
> at
> org.apache.hadoop.fs.RawLocalFileSystem.initialize(RawLocalFileSystem.java:48)
> at
> org.apache.hadoop.fs.LocalFileSystem.initialize(LocalFileSystem.java:47)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:433)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:301)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:378)
> at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:461)
> at
> org.apache.hadoop.fs.LocalDirAllocator.createTmpFileForWrite(LocalDirAllocator.java:200)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.createTmpFileForWrite(S3AFileSystem.java:572)
> at
> org.apache.hadoop.fs.s3a.S3ADataBlocks$DiskBlockFactory.create(S3ADataBlocks.java:811)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.createBlockIfNeeded(S3ABlockOutputStream.java:190)
> at
> org.apache.hadoop.fs.s3a.S3ABlockOutputStream.<init>(S3ABlockOutputStream.java:168)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:778)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1026)
> at
> org.apache.oozie.action.hadoop.LauncherMapperHelper.setupLauncherInfo(LauncherMapperHelper.java:156)
> at
> org.apache.oozie.action.hadoop.JavaActionExecutor.createLauncherConf(JavaActionExecutor.java:1033)
> ... 12 more
> {noformat}
> Does anybody has any idea how could we modify the RawLocalFileSystem class to
> make it a bit less strict?
> Thank you for the repro wf to Soumitra Sulav.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
