Hi,
If I understand correctly, you can read data from ES (through es-hadoop) but you cannot write to it - am I correct? Can
you confirm that you are using the latest es-hadoop, namely 1.3.0.M3?
How big is the JSON file you are trying to index? Do you see any activity in
the console?
There are various ways in which you can monitor activity - in ES you can monitor the console or use Marvel [1], in
ES-hadoop you can enable logging [3] and see how the job progresses. Try starting with a small file to have a short
feedback loop and once things get ironed out, try your actual desired file.
A few notes:
- In general we recommend using the latest stable version of Elasticsearch - 0.90.2 is quite old and unless you have a
strong reason to stay on it, I highly recommend upgrading to 1.1.1 or, in the worst case scenario, ES 0.90.13.
- you have repeating code - you set "es.nodes" and "es.port" twice
- you are copying the local file to the destination filesystem (presumably HDFS) which is okay but typically this is
done outside the job launch
- if you are using Hadoop MRv2, consider switching to MRv1. es-hadoop supports both modes but the latter is easier to
use and still the one the vendors recommend by default
- since you are using Hadoop 2 from Hortonworks, you might want to upgrade to
their latest HDP (2.1) release.
Hope this helps,
[1] http://www.elasticsearch.org/overview/marvel/
[2]
http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/logging.html
On 4/24/14 7:30 AM, Chetana wrote:
I am using elasticsearch-hadoop plugin
(https://github.com/elasticsearch/elasticsearch-hadoop) and trying to index some
documents. Iam using elasticsearch version 0.90.2 and Hadoop Hortonworks 2.2.0.
The search functionality works fine, but
while indexing application hangs
https://github.com/elasticsearch/elasticsearch-hadoop
The json file location is passed as a command line arguemnt and below is the
indexing code snippet
Configuration conf = new Configuration();
conf.setBoolean("mapred.map.tasks.speculative.execution", false);
conf.setBoolean("mapred.reduce.tasks.speculative.execution", false);
conf.setInt("mapred.min.split.size",40);
conf.set("es.resource", "test/test");
conf.set("es.nodes", "localhost");
conf.set("es.port", "9200");
conf.set("es.input.json", "yes");
conf.set("es.nodes", "localhost");
conf.set("es.port", "9200");
Job job = Job.getInstance(conf);
job.setMapperClass(Mapper.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(EsOutputFormat.class);
job.setMapOutputKeyClass(LongWritable.class);
job.setMapOutputValueClass(Text.class);
Path jarPath = new Path(args[0]);
FileSystem fs = FileSystem.get(conf);
Path dst = new Path(fs.getHomeDirectory(), jarPath.getName());
fs.copyFromLocalFile(false, true, jarPath, dst);
FileInputFormat.setInputPaths(job, dst);
job.waitForCompletion(true);
Am I missing anything, pls help
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
elasticsearch+unsubscr...@googlegroups.com
<mailto:elasticsearch+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/15b0d1e9-e258-4fd4-b7bc-d7b81596fc81%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53589A66.4000805%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
