Seems like this would a good additional strategy, something like
ENVELOPE_PARSED? Any thoughts on a good name?
On Thu, 25 Apr 2019 at 16:20, Otto Fowler <ottobackwa...@gmail.com> wrote:
> So, the enveloped message doesn’t support getting an already parsed json
> object from the enveloped json, we would have to do some work to support
> this, Even if we _could_ wrangle it in there now, from what I can see we
> would still have to serialize to bytes to pass to the actual parser and
> that would be inefficient.
> Can you open a jira with the information you provided?
>
>
>
> On April 25, 2019 at 11:12:38, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> Raw message in this case assumes that the raw message is a String embedded
> in the json field that you supply, not a nested json object, so it is
> looking for
>
>
> “_source” : “some other embedded string of some format like syslog in json”
>
> There are other message strategies, but I’m not sure they would work in
> this instance. I’ll keep looking. hopefully someone more familiar will
> jump in.
>
>
> On April 25, 2019 at 10:48:06, stephane.d...@orange.com (
> stephane.d...@orange.com) wrote:
>
> Hello,
>
>
>
> I’m trying to load some JSON data which has the following structure (this
> is a sample):
>
>
>
> {
>
> "_index": "indexing",
>
> "_type": "Event",
>
> "_id": "AWAkTAefYn0uCUpkHmCy",
>
> "_score": 1,
>
> "_source": {
>
> "dst": "127.0.0.1",
>
> "devTimeEpoch": "1512437340000",
>
> "dstPort": "0",
>
> "srcPort": "80",
>
> "src": "194.51.198.185"
>
> }
>
> }
>
>
>
> In my file, everything is on the same line. My parser config is the
> following:
>
>
>
> {
>
> "parserClassName": "org.apache.metron.parsers.json.JSONMapParser",
>
> "filterClassName": null,
>
> "sensorTopic": "my_topic",
>
> "outputTopic": null,
>
> "errorTopic": null,
>
> "writerClassName": null,
>
> "errorWriterClassName": null,
>
> "readMetadata": true,
>
> "mergeMetadata": true,
>
> "numWorkers": 2,
>
> "numAckers": null,
>
> "spoutParallelism": 1,
>
> "spoutNumTasks": 1,
>
> "parserParallelism": 2,
>
> "parserNumTasks": 2,
>
> "errorWriterParallelism": 1,
>
> "errorWriterNumTasks": 1,
>
> "spoutConfig": {},
>
> "securityProtocol": null,
>
> "stormConfig": {},
>
> "parserConfig": {
>
> },
>
> "fieldTransformations": [
>
> {
>
> "transformation":"RENAME",
>
> "config": {
>
> "dst": "ip_dst_addr",
>
> "src": "ip_src_addr",
>
> "srcPort": "ip_src_port",
>
> "dstPort": "ip_dst_port",
>
> "devTimeEpoch": "timestamp"
>
> }
>
> }
>
> ],
>
> "cacheConfig": {},
>
> "rawMessageStrategy": "ENVELOPE",
>
> "rawMessageStrategyConfig": {
>
> "messageField": "_source"
>
> }
>
> }
>
>
>
> But in Storm I get the following errors:
>
>
>
> 2019-04-25 16:45:22.225 o.a.s.d.executor Thread-5-parserBolt-executor[8 8]
> [ERROR]
>
> java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to
> java.lang.String
>
> at
> org.apache.metron.common.message.metadata.EnvelopedRawMessageStrategy.get(EnvelopedRawMessageStrategy.java:78)
> ~[stormjar.jar:?]
>
> at
> org.apache.metron.common.message.metadata.RawMessageStrategies.get(RawMessageStrategies.java:54)
> ~[stormjar.jar:?]
>
> at
> org.apache.metron.common.message.metadata.RawMessageUtil.getRawMessage(RawMessageUtil.java:55)
> ~[stormjar.jar:?]
>
> at
> org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:251)
> [stormjar.jar:?]
>
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.1050-37.jar:1.1.0.2.6.5.1050-37]
>
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
>
>
>
>
>
> How can I debug this?
>
>
>
> Thanks
>
>
>
> Stéphane
>
> _________________________________________________________________________________________________________________________
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
> ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and its attachments may contain confidential or privileged
> information that may be protected by law;
> they should not be distributed, used or copied without authorisation.
> If you have received this email in error, please notify the sender and delete
> this message and its attachments.
> As emails may be altered, Orange is not liable for messages that have been
> modified, changed or falsified.
> Thank you.
>
>
--
--
simon elliston ball
@sireb
