[
https://issues.apache.org/jira/browse/OOZIE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryota Egashira updated OOZIE-2246:
----------------------------------
Description:
when har file doesn't exist, FileSystem.createFileSystem in
HadoopAccessorService throw exception, failing CoordActionInputCheck.
thus, even thought there are other data dependencies which already exist, it is
not reflected on DB.
coordinator job cannot start until the har file becomes available anyway, and
once available, this error doesn't happen, so basic functionality is fine, but
it's misleading.
{code}
2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542
[pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-]
exception callable [coord_action_input], E1021: Coord Action Input Check Error:
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured:
[Invalid path for the Har Filesystem. No index file in
har://aaaa:8020/data/2014031322/archive.har]
org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception
occured: [Invalid path for the Har Filesystem. No index file in
har://aaaa:8020/data/2014031322/archive.har]
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException:
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured:
[Invalid path for the Har Filesystem. No index file in
har://aaaa:8020/data/2014031322/archive.har]
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
... 6 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception
occured: [Invalid path for the Har Filesystem. No index file in
har://aaaa:8020/data/2014031322/archive.har]
at
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
... 10 more
Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index
file in har://aaaa:8020/data/2014031322/archive.har
at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
at
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
{code}
was:
when har file doesn't exist, FileSystem.createFileSystem in
HadoopAccessorService throw exception, failing CoordActionInputCheck.
thus, even thought there are other data dependencies which already exist, it is
not reflected on DB.
coordinator job cannot start until the har file becomes available anyway, and
once available, this error doesn't happen, so basic functionality is fine, but
it's misleading.
{code}
2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542
[pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-]
exception callable [coord_action_input], E1021: Coord Action Input Check Error:
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured:
[Invalid path for the Har Filesystem. No index file in
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception
occured: [Invalid path for the Har Filesystem. No index file in
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
at org.apache.oozie.command.XCommand.call(XCommand.java:280)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.IOException:
org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured:
[Invalid path for the Har Filesystem. No index file in
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
... 6 more
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception
occured: [Invalid path for the Har Filesystem. No index file in
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har]
at
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
at
org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
... 10 more
Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index
file in
har://hdfs-dilithiumred-nn1.red.ygrid.yahoo.com:8020/projects/tpoozie/post_tp_rmx_joined_click_trim_har/hourly/data/2014031322/archive.har
at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
at
org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
at
org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
{code}
> CoordinatorInputCheckCommand does not behave properly when har file is one of
> data dependency and doesn't exist
> ---------------------------------------------------------------------------------------------------------------
>
> Key: OOZIE-2246
> URL: https://issues.apache.org/jira/browse/OOZIE-2246
> Project: Oozie
> Issue Type: Bug
> Reporter: Ryota Egashira
> Assignee: Ryota Egashira
> Attachments: OOZIE-2246.patch
>
>
> when har file doesn't exist, FileSystem.createFileSystem in
> HadoopAccessorService throw exception, failing CoordActionInputCheck.
> thus, even thought there are other data dependencies which already exist, it
> is not reflected on DB.
> coordinator job cannot start until the har file becomes available anyway, and
> once available, this error doesn't happen, so basic functionality is fine,
> but it's misleading.
> {code}
> 2014-03-13 22:00:00,051 WARN CallableQueueService$CallableWrapper:542
> [pool-2-thread-288] - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-]
> exception callable [coord_action_input], E1021: Coord Action Input Check
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception
> occured: [Invalid path for the Har Filesystem. No index file in
> har://aaaa:8020/data/2014031322/archive.har]
> org.apache.oozie.command.CommandException: E1021: Coord Action Input Check
> Error: org.apache.oozie.service.HadoopAccessorException: E0902: Exception
> occured: [Invalid path for the Har Filesystem. No index file in
> har://aaaa:8020/data/2014031322/archive.har]
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:182)
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:67)
> at org.apache.oozie.command.XCommand.call(XCommand.java:280)
> at
> org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.io.IOException:
> org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured:
> [Invalid path for the Har Filesystem. No index file in
> har://aaaa:8020/data/2014031322/archive.har]
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:493)
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:459)
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:429)
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:259)
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.execute(CoordActionInputCheckXCommand.java:132)
> ... 6 more
> Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception
> occured: [Invalid path for the Har Filesystem. No index file in
> har://aaaa:8020/data/2014031322/archive.har]
> at
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:430)
> at
> org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:134)
> at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:99)
> at
> org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:488)
> ... 10 more
> Caused by: java.io.IOException: Invalid path for the Har Filesystem. No index
> file in har://aaaa:8020/data/2014031322/archive.har
> at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:139)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2160)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:303)
> at
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:422)
> at
> org.apache.oozie.service.HadoopAccessorService$2.run(HadoopAccessorService.java:420)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1284)
> at
> org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:420)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
