Hi, I am trying to make Oozie work seemless with S3. I have managed, faffing a little bit with Amazon EMR, to make Oozie load the workflow (and coordinator) xml from S3. However, I am planning to move Oozie on his own machine, so I cannot rely on the changes I have made to work with EMR. [1]
I have found this: https://issues.apache.org/jira/browse/OOZIE-426 and followed the instruction as described in the comment section [2]. However, I always receive an "Internal server error" when I "dryrun" a workflow. Digging into the oozie.log file, I have found the following stack trace: 2014-10-07 14:04:00,016 ERROR V1JobsServlet:536 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST http://localhost:11000/oozie/v2/jobs?action=dryrun] error, null java.lang.NullPointerException at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:433) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398) at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:370) at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:192) at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:96) at javax.servlet.http.HttpServlet.service(HttpServlet.java:643) at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:286) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:392) at org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:84) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:745) S3 credentials have been added to the core-site.xml file in the oozie/conf directory, as described in https://wiki.apache.org/hadoop/AmazonS3 I am using Oozie 4.0.1 compiled to work with Hadoop 2.4.0. Thanks for your help, Davide [1] my work on this is available at https://github.com/davideanastasia/emr-oozie-sample/blob/master/config/config-oozie.sh [2] https://issues.apache.org/jira/browse/OOZIE-426?focusedCommentId=13899687&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13899687 -- @BlisMedia <http://twitter.com/BlisMedia> www.blismedia.com <http://blismedia.com> This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of BlisMedia Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 3rd Floor, 101 New Cavendish St, London, W1W 6XH, United Kingdom. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
