Hello, Thanks Mohammad.
Almost there. I had to changed the hive-site.xml from what was installed. hive.metastore.client.connect.retry.delay with had 1s to 1. hive.metastore.client.socket.timeout from 600s to 600. The oozie 4.1.0 hive example almost works. The hive table "test" does get created with the input data. Oozie to hive. However even though the script was successful in creating and load the test table: 07093549669-oozie-oozi-W] ACTION[0000009-150407093549669-oozie-oozi-W@hive-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [1] 2015-04-07 12:31:22,240 INFO ActionEndXCommand:541 - SERVER[localhost.localdomain] USER[oozie] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000009-15040 Hmmm. Not sure at the moment. Thanks, Gazza From: Mohammad Islam [mailto:[email protected]] Sent: Tuesday, April 07, 2015 11:58 AM To: Gary Clark; [email protected] Subject: Re: Oozie Hive job running continuously Hi Gazza, Not sure what causes this. Did you able to run any simple hive command? even the example that comes with oozie-examples? may be related: Loading data to MySQL table from Hive using Sqoop/Oozie<http://stackoverflow.com/questions/23663978/loading-data-to-mysql-table-from-hive-using-sqoop-oozie> [image]<http://stackoverflow.com/questions/23663978/loading-data-to-mysql-table-from-hive-using-sqoop-oozie> Loading data to MySQL table from Hive using Sqoop/Oozie<http://stackoverflow.com/questions/23663978/loading-data-to-mysql-table-from-hive-using-sqoop-oozie> I am facing some grave issue (tried unsuccessful 196 times) while loading data using Sqoop (sqoop command inside Oozie) to MySQL table. If there is only one column... View on stackoverflow.com<http://stackoverflow.com/questions/23663978/loading-data-to-mysql-table-from-hive-using-sqoop-oozie> Preview by Yahoo Regards, Mohammad On Tuesday, April 7, 2015 8:13 AM, Gary Clark <[email protected]<mailto:[email protected]>> wrote: Hello, Running the oozie job I hit strange problem: Looking at the oozie log I see the below: 2015-04-07 09:49:16,109 WARN JobSubmitter:150 - SERVER[localhost.localdomain] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2015-04-07 09:49:16,117 WARN JobSubmitter:259 - SERVER[localhost.localdomain] No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2015-04-07 09:49:16,664 INFO HiveActionExecutor:541 - SERVER[localhost.localdomain] USER[oozie] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000001-150407093549669-oozie-oozi-W] ACTION[0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] checking action, hadoop job ID [job_1428416855230_0002] status [RUNNING] 2015-04-07 09:49:16,667 INFO ActionStartXCommand:541 - SERVER[localhost.localdomain] USER[oozie] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000001-150407093549669-oozie-oozi-W] ACTION[0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] [***0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>***]Action status=RUNNING 2015-04-07 09:49:16,667 INFO ActionStartXCommand:541 - SERVER[localhost.localdomain] USER[oozie] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000001-150407093549669-oozie-oozi-W] ACTION[0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] [***0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>***]Action updated in DB! 2015-04-07 09:49:36,803 INFO CallbackServlet:541 - SERVER[localhost.localdomain] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-150407093549669-oozie-oozi-W] ACTION[0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] callback for action [0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] 2015-04-07 09:49:39,639 INFO HiveActionExecutor:541 - SERVER[localhost.localdomain] USER[oozie] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000001-150407093549669-oozie-oozi-W] ACTION[0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] action completed, external ID [job_1428416855230_0002] 2015-04-07 09:49:40,004 WARN HiveActionExecutor:544 - SERVER[localhost.localdomain] USER[oozie] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000001-150407093549669-oozie-oozi-W] ACTION[0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], main() threw exception, java.lang.NumberFormatException: For input string: "1s" 2015-04-07 09:49:40,006 WARN HiveActionExecutor:544 - SERVER[localhost.localdomain] USER[oozie] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000001-150407093549669-oozie-oozi-W] ACTION[0000001-150407093549669-oozie-oozi-W@hive-node<mailto:0000001-150407093549669-oozie-oozi-W@hive-node>] Launcher exception: java.lang.NumberFormatException: For input string: "1s" java.lang.RuntimeException: java.lang.NumberFormatException: For input string: "1s" at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:323) at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:284) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.NumberFormatException: For input string: "1s" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:492) at java.lang.Integer.parseInt(Integer.java:527) at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1134) at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1211) at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1220) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:58) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340) Im seeing a strange error. I'm running hiveServer2 and metastore. Ideas on how to remedy this would be appreciated. Thanks, Gazza -----Original Message----- From: Gary Clark [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, April 07, 2015 8:25 AM To: [email protected]<mailto:[email protected]>; Mohammad Islam Subject: RE: Oozie Hive job running continuously Totallly agree. I will post on this thread. -----Original Message----- From: Mohammad Islam [mailto:[email protected]<mailto:[email protected]>] Sent: Monday, April 06, 2015 4:59 PM To: [email protected]<mailto:[email protected]> Subject: Re: Oozie Hive job running continuously Please share with us, how did you resolve the issue. It will hep others in future. You should include "<job-xml>${hiveSiteXML}</job-xml>" after <name-node> or before <configurations>? It has some sequential dependencies.More at :Oozie - Regards,Mohammad On Monday, April 6, 2015 2:41 PM, Gary Clark <[email protected]<mailto:[email protected]>> wrote: Thanks Islam. I made a wee bit of progress. In that I did get a succeed job. However no table was created. I am using 4.1.0. I perform the below: 1) ./hive --service metastore This should then connects to the local derby database. 2) It then fails with the below: <workflow-app xmlns="uri:oozie:workflow:0.2" name="hive-wf"> <start to="hive-node"/> <action name="hive-node"> <hive xmlns="uri:oozie:hive-action:0.2"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>mapred.job.queue.name</name> <value>${queueName}</value> </property> </configuration> <job-xml>${hiveSiteXML}</job-xml> <script>script.q</script> </hive> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Hive failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app> Error: E0701 : E0701: XML schema error, cvc-complex-type.2.4.a: Invalid content was found starting with element 'job-xml'. One of '{"uri:oozie:hive-action:0.2":script}' is expected. It does not like the "job-xml" Not sure what version to use. Please advise. I need to use job-xml to specify the hive-site.xml. Thanks, Gazza -----Original Message----- From: Mohammad Islam [mailto:[email protected]<mailto:[email protected]>] Sent: Monday, April 06, 2015 12:08 PM To: [email protected]<mailto:[email protected]> Subject: Re: Oozie Hive job running continuously Hi Gazza,You seems to identify the problem correctly.You have some resource shortage. You can check RM UI to see how much resources are available and already used.You may consider to use uber job for launcher. Also try to reduce the container memory for Launcher. Alternatively, you can increase the number of nodes in your cluster. Regards,Mohammad On Monday, April 6, 2015 7:48 AM, Gary Clark <[email protected]<mailto:[email protected]>> wrote: Hello, I am hoping this is a common problem. I am running a single node cluster (1 machine) with Hadoop 2.6.0 and OOZIE 4.1.0 with hiveServer2 running in the background. I am executing the hive work flow example provided in the examples: What I am seeing is the below in the hadoop logs: 2015-04-06 08:57:01,475 INFO [IPC Server handler 0 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1428328525963_0001_m_000002 asked for a task 2015-04-06 08:57:01,476 INFO [IPC Server handler 0 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1428328525963_0001_m_000002 given task: attempt_1428328525963_0001_m_000000_0 2015-04-06 08:57:09,052 INFO [IPC Server handler 29 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 2015-04-06 08:57:39,354 INFO [IPC Server handler 27 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 2015-04-06 08:58:09,474 INFO [IPC Server handler 29 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 2015-04-06 08:58:36,866 INFO [IPC Server handler 25 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 2015-04-06 08:59:07,026 INFO [IPC Server handler 25 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 2015-04-06 08:59:37,142 INFO [IPC Server handler 26 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 2015-04-06 09:00:07,278 INFO [IPC Server handler 28 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 2015-04-06 09:00:37,389 INFO [IPC Server handler 26 on 59454] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1428328525963_0001_m_000000_0 is : 1.0 It looks like a resource constraint. My yarn-site.xml: <?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>5120</value> <description>Amount of physical memory, in MB, that can be allocated for containers.</description> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>512</value> <description> The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this won't take effect, and the specified value will get allocated at minimum. </description> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>2048</value> <description> The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won't take effect, and will get capped to this value. </description> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property> <configuration> Anybody seen this? I am running this on a virtual machine. Much Appreciated, Gazza
