Re: No response from ignite job tracker
Thanks Ilya. The reason I used that address is because the value in the example config provided with the ignite-hadoop package is: mapreduce.jobtracker.address localhost:11211 I tried changing it to just "local" which I know is the default, but I get the following error: 2018-08-14 14:47:13,714 INFO [main] mapreduce.Cluster (Cluster.java:initialize(113)) - Failed to use org.apache.ignite.hadoop.mapreduce.IgniteHadoopClientProtocolProvider due to error: java.io.IOException: Local execution mode is not supported, please point mapreduce.jobtracker.address to real Ignite nodes. I also tried changing it to just my server name with no port reference like: mapreduce.jobtracker.address myserver.com and leaving it the value in the example ignite-hadoop configs: mapreduce.jobtracker.address localhost:11211 In both of those cases, it shows the url as N/A: 2018-08-14 15:09:24,285 INFO [main] mapreduce.Job (Job.java:submit(1294)) - The url to track the job: N/A The description for mapreduce.jobtracker.address in the hadoop documentation is: "The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task." What port does ignite use for this if it isn't 11211? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: No response from ignite job tracker
Hello! You should not use curl to test 11211 port since it's not an HTTP port. I don't think that Ignite binary client protocol supports whatever mapreduce.jobtracker.address expects. Have you tried "local"? Regards, -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: No response from ignite job tracker
Still looking for any suggestions on this. Anyone have any ideas for next steps to troubleshoot? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: No response from ignite job tracker
I can see messages in the ignite logs when I run my mapreduce job: [2018-08-06 10:15:56,914][WARN ][pub-#23817][HadoopJobTracker] ignite.job.shared.classloader job property is set to true; please disable it if job tasks rely on mutable static state. [2018-08-06 10:15:58,303][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-SETUP-0-0-#23820][FileOutputCommitter] File Output Committer Algorithm version is 1 [2018-08-06 10:15:58,303][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-SETUP-0-0-#23820][FileOutputCommitter] FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false [2018-08-06 10:15:58,474][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-MAP-0-0-#23829][LineRecordReader] Found UTF-8 BOM and skipped it [2018-08-06 10:15:58,755][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-REDUCE-0-0-#23831][FileOutputCommitter] File Output Committer Algorithm version is 1 [2018-08-06 10:15:58,756][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-REDUCE-0-0-#23831][FileOutputCommitter] FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false [2018-08-06 10:15:58,884][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-REDUCE-0-0-#23831][FileOutputCommitter] Saved output of task 'attempt_e93a350c-a588-42ee-883a-f0efec20aaed_0001_r_00_0' to hdfs://ignitehdp/tmp/output/_temporary/0/task_e93a350c-a588-42ee-883a-f0efec20aaed_0001_r_00 [2018-08-06 10:15:58,910][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-COMMIT-0-0-#23832][FileOutputCommitter] File Output Committer Algorithm version is 1 [2018-08-06 10:15:58,910][INFO ][Hadoop-task-e93a350c-a588-42ee-883a-f0efec20aaed_1-COMMIT-0-0-#23832][FileOutputCommitter] FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false I can also telnet from my laptop where I opening a browser to the server that is running ignite on port 11211: $ telnet myserver.com 11211 Trying xxx.xxx.xxx.xxx... Connected to myserver.com. Escape character is '^]'. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: No response from ignite job tracker
Hi, Are you sure that these requests reach the server? Do you have anything suspicious in Ignite logs? I'd recommend to check them at first. Evgenii 2018-08-03 23:42 GMT+03:00 engrdean : > I am successfully running mapreduce jobs using Ignite but I haven't been > able > to use the Ignite job tracker. I'm running my job like this: > > $ /bin/hadoop --config ~/ignite/ignite_conf jar wc.jar WordCount > /user/ignite/ /tmp/output > 2018-08-03 15:09:32,751 WARN [main] mapreduce.JobResourceUploader > (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option > parsing not performed. Implement the Tool interface and execute your > application with ToolRunner to remedy this. > 2018-08-03 15:09:32,960 INFO [main] input.FileInputFormat > (FileInputFormat.java:listStatus(283)) - Total input paths to process : 1 > 2018-08-03 15:09:33,016 INFO [main] mapreduce.JobSubmitter > (JobSubmitter.java:submitJobInternal(198)) - number of splits:1 > 2018-08-03 15:09:33,099 INFO [main] mapreduce.JobSubmitter > (JobSubmitter.java:printTokens(287)) - Submitting tokens for job: > job_a217f633-f2d7-4255-adad-be19b9cda204_0003 > 2018-08-03 15:09:35,906 INFO [main] mapreduce.Job (Job.java:submit(1294)) > - > The url to track the job: N/A > 2018-08-03 15:09:35,906 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1339)) - Running job: > job_a217f633-f2d7-4255-adad-be19b9cda204_0003 > 2018-08-03 15:09:37,230 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1360)) - Job > job_a217f633-f2d7-4255-adad-be19b9cda204_0003 running in uber mode : false > 2018-08-03 15:09:37,232 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0% > 2018-08-03 15:09:38,531 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 100% > 2018-08-03 15:09:38,548 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1378)) - Job > job_a217f633-f2d7-4255-adad-be19b9cda204_0003 completed successfully > 2018-08-03 15:09:38,564 INFO [main] mapreduce.Job > (Job.java:monitorAndPrintJob(1385)) - Counters: 0 > > As you can see, it lists the job tracker url as "N/A". I believe I have > correctly set the url in the mapred-site.xml: > > $ grep -C 2 mapreduce.jobtracker.address ignite_conf/mapred-site.xml > > > mapreduce.jobtracker.address > myserver.com:11211 > > > However, I'm getting an empty reply from the server: > > $ curl http://myserver.com:11211 > curl: (52) Empty reply from server > > I can see the port is up: > > $ netstat -luptn | grep 11211 > (Not all processes could be identified, non-owned process info > will not be shown, you would have to be root to see it all.) > tcp0 0 0.0.0.0:11211 0.0.0.0:* > LISTEN > 11709/java > > > What am I missing here? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
No response from ignite job tracker
I am successfully running mapreduce jobs using Ignite but I haven't been able to use the Ignite job tracker. I'm running my job like this: $ /bin/hadoop --config ~/ignite/ignite_conf jar wc.jar WordCount /user/ignite/ /tmp/output 2018-08-03 15:09:32,751 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2018-08-03 15:09:32,960 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(283)) - Total input paths to process : 1 2018-08-03 15:09:33,016 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(198)) - number of splits:1 2018-08-03 15:09:33,099 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(287)) - Submitting tokens for job: job_a217f633-f2d7-4255-adad-be19b9cda204_0003 2018-08-03 15:09:35,906 INFO [main] mapreduce.Job (Job.java:submit(1294)) - The url to track the job: N/A 2018-08-03 15:09:35,906 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job: job_a217f633-f2d7-4255-adad-be19b9cda204_0003 2018-08-03 15:09:37,230 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job job_a217f633-f2d7-4255-adad-be19b9cda204_0003 running in uber mode : false 2018-08-03 15:09:37,232 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0% 2018-08-03 15:09:38,531 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 100% reduce 100% 2018-08-03 15:09:38,548 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1378)) - Job job_a217f633-f2d7-4255-adad-be19b9cda204_0003 completed successfully 2018-08-03 15:09:38,564 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 0 As you can see, it lists the job tracker url as "N/A". I believe I have correctly set the url in the mapred-site.xml: $ grep -C 2 mapreduce.jobtracker.address ignite_conf/mapred-site.xml mapreduce.jobtracker.address myserver.com:11211 However, I'm getting an empty reply from the server: $ curl http://myserver.com:11211 curl: (52) Empty reply from server I can see the port is up: $ netstat -luptn | grep 11211 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp0 0 0.0.0.0:11211 0.0.0.0:* LISTEN 11709/java What am I missing here? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
