Hi,
How do you pass the json to es-hadoop? Do you have an example? By the way, you can enable TRACE logging on
org.elasticsearch.hadoop and see everything that es-hadoop does, including the data that goes over the wire.
My guess is that the conversion of logs to JSON creates some extra artifacts which are later on interpreted as Writable
object (instead of raw JSON) by ES Hadoop.
Make sure you tell es-hadoop that its source it's json (through es.input.json
set to true).
The logs will likely confirm (or not) the above :)
Cheers,
On 3/19/14 11:14 PM, Brian Stempin wrote:
Hi List,
I have an ES cluster that takes in some data from our logs. We use Hadoop to
parse the individual log entries into JSON
strings, which does a bulk insert using ES's output format. For whatever
reason, ES attempts to parse base64 strings as
a dates and fails. Here's a line from one of my Hadoop logs:
java.lang.IllegalStateException: Found unrecoverable error [Bad Request(400) -
MapperParsingException[failed to parse [csUriParams.d]]; nested:
MapperParsingException[failed to parse date field [REDACTED BASE64 STRING], tried both
date format [dateOptionalTime], and timestamp number with locale []]; nested:
IllegalArgumentException[Invalid format:
"Y2lkPURFJml0ZW1zPWE2NTJjLXgxZTFj..."]; ]; Bailing out..
at
org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(RestClient.java:145)
at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:120)
at
org.elasticsearch.hadoop.rest.RestRepository.sendBatch(RestRepository.java:147)
<SNIP>
csUriParams.d does not appear in my mapping, so I never explicitly asked for it
to be treated as a date.
Any idea why ES is trying to treat it as a date?
Thanks,
Brian
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to
[email protected]
<mailto:[email protected]>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/49e5fe0b-cec3-4914-b8d6-99440dd5fb69%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/49e5fe0b-cec3-4914-b8d6-99440dd5fb69%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/532A0D9C.7010401%40gmail.com.
For more options, visit https://groups.google.com/d/optout.
