Hi, I've been trying to install Oozie 4.2.0 on a EC2 machine for a few days. I have managed to compile Oozie using -Phadoop-2 and -Puber and I create my war adding ext, jets3t 0.9.0 and hadoop-lzo in my libext/ folder.
Everything seems to be going well, and I can interact with an EMR cluster (Hadoop 2.4.0), however, I cannot access S3 (where most of my inputs are). I have changed the core-site.xml in the hadoop-conf folder in order to add properties for fs.s3n.awsAccessKeyId and fs.s3n.awsSecretAccessKey. When I launch my job.properties (both coordinator.xml and workflow.xml are on HDFS), I constantly get: davide@...:~/test-oozie$ /opt/oozie-4.2.0/bin/oozie job -oozie http://localhost:11000/oozie/ -config job.properties -dryrun [...] Error: E1003 : E1003: Invalid coordinator application attributes, AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively). In the oozie-error.log I get: Caused by: org.apache.oozie.CoordinatorEngineException: E1003: Invalid coordinator application attributes, AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting t he fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively). at org.apache.oozie.CoordinatorEngine.dryRunSubmit(CoordinatorEngine.java:564) at org.apache.oozie.servlet.V1JobsServlet.submitCoordinatorJob(V1JobsServlet.java:228) ... 25 more Caused by: org.apache.oozie.command.CommandException: E1003: Invalid coordinator application attributes, AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively). at org.apache.oozie.command.coord.CoordSubmitXCommand.submitJob(CoordSubmitXCommand.java:269) at org.apache.oozie.command.coord.CoordSubmitXCommand.submit(CoordSubmitXCommand.java:203) at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:82) at org.apache.oozie.command.SubmitTransitionXCommand.execute(SubmitTransitionXCommand.java:30) at org.apache.oozie.command.XCommand.call(XCommand.java:286) at org.apache.oozie.CoordinatorEngine.dryRunSubmit(CoordinatorEngine.java:561) ... 26 more Caused by: java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey proper ties (respectively). at org.apache.hadoop.fs.s3.S3Credentials.initialize(S3Credentials.java:70) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103) at org.apache.hadoop.fs.s3native.$Proxy28.initialize(Unknown Source) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:272) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2397) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:488) at org.apache.oozie.service.HadoopAccessorService$4.run(HadoopAccessorService.java:486) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.oozie.service.HadoopAccessorService.createFileSystem(HadoopAccessorService.java:486) at org.apache.oozie.dependency.FSURIHandler.getFileSystem(FSURIHandler.java:166) at org.apache.oozie.dependency.FSURIHandler.exists(FSURIHandler.java:100) at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.pathExists(CoordActionInputCheckXCommand.java:579) at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkListOfPaths(CoordActionInputCheckXCommand.java:550) at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkResolvedUris(CoordActionInputCheckXCommand.java:520) at org.apache.oozie.command.coord.CoordActionInputCheckXCommand.checkInput(CoordActionInputCheckXCommand.java:351) at org.apache.oozie.command.coord.CoordCommandUtils.dryRunCoord(CoordCommandUtils.java:573) at org.apache.oozie.command.coord.CoordCommandUtils.materializeOneInstance(CoordCommandUtils.java:541) at org.apache.oozie.command.coord.CoordMaterializeTransitionXCommand.materializeActions(CoordMaterializeTransitionXCommand.java:485) at org.apache.oozie.command.coord.CoordSubmitXCommand.getDryRun(CoordSubmitXCommand.java:316) at org.apache.oozie.command.coord.CoordSubmitXCommand.submitJob(CoordSubmitXCommand.java:248) ... 31 more What am I doing wrong? Regards, Davide -- This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of BlisMedia Ltd, a company registered in England and Wales with registered number 06455773. Its registered office is 3rd Floor, 101 New Cavendish St, London, W1W 6XH, United Kingdom. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.
