Such a careless mistake. Thanks for the help! :) Regards, Mustafa Akmal On Feb 27 2019, at 7:10 pm, Stefan Kupstaitis-Dunkler <[email protected]> wrote: > Hi Mustafa, > > Looking at your parser config, you've set these values within the > "rawMessageStrategyConfig" > > "readMetadata": "true", > "mergeMetadata": "true" > > > However, there they will be ignored. Have a look at the same properties just > below the "errorWriterClassName" property. Both of them are set to false. > You can remove those properties from the "rawMessageStrategyConfig" section > and set them true in the upper part of your config! I'm confident that will > make it work for you! > > Best, > Stefan > > > On Wed, Feb 27, 2019 at 6:04 AM Mustafa Akmal <[email protected] > (mailto:[email protected])> wrote: > > Hi Stefan > > Thanks for the quick response. > > I am using Metron version 0.6.0.1.7.1.0 > > (https://link.getmailspring.com/link/[email protected]/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D) > > > > Here is my parser config > > { > > "parserClassName": "org.apache.metron.parsers.csv.CSVParser", > > "filterClassName": null, > > "sensorTopic": "test_syslog", > > "outputTopic": null, > > "errorTopic": null, > > "writerClassName": null, > > "errorWriterClassName": null, > > "readMetadata": false, > > "mergeMetadata": false, > > "numWorkers": null, > > "numAckers": null, > > "spoutParallelism": 1, > > "spoutNumTasks": 1, > > "parserParallelism": 1, > > "parserNumTasks": 1, > > "errorWriterParallelism": 1, > > "errorWriterNumTasks": 1, > > "spoutConfig": {}, > > "securityProtocol": null, > > "stormConfig": {}, > > "parserConfig": { > > "columns": { > > "Id": 0, > > "Name": 1, > > "Age": 2 > > } > > }, > > "fieldTransformations": [], > > "cacheConfig": {}, > > "rawMessageStrategy": "ENVELOPE", > > "rawMessageStrategyConfig": { > > "messageField": "data", > > "readMetadata": "true", > > "mergeMetadata": "true" > > } > > } > > > > I have been trying to do this for a couple of days. Tried it on multiple > > different parsers after studying documentation. At first I was using a > > parser chaining usecase. But now I wrote a simple parser to test the > > problem. Set the readMetadata and mergeMetadata fields to true just in > > case. It just doesn't work. > > Regards, > > Mustafa Akmal > > > > On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler <[email protected] > > (mailto:[email protected])> wrote: > > > Hi Mustafa, > > > > > > can you verify if the "mergeMetaData" property in the parser json is set > > > to "true". > > > If this property is set to false, other fields won't be merged as meta > > > data. > > > > > > If this property is not set, it should default to true for the message > > > strategy "ENVELOPE". > > > > > > Any other behaviour is probably a bug. > > > > > > Also verify if you set "rawMessageStrategy" to "true" for the same > > > reasons. > > > > > > > > > Best, > > > > > > Stefan > > > > > > > > > On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <[email protected] > > > (mailto:[email protected])> wrote: > > > > Hello > > > > I am using a CSV parser. I have the following log > > > > {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"} > > > > > > > > Now I have set the raw message strategy to 'ENVELOPE' and the > > > > messageField to 'data' > > > > However after the record is indexed in elasticsearch, the parser does > > > > parse the value inside data but it does not get the additional fields > > > > as shown in the original log that is 'AdditionField' and > > > > 'AdditionField2'. What am I doing wrong? Can anyone help? > > > > Thanks! > > > > > > > > > > > > Virus-free. www.avg.com > > > > (http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient) > > > > > > -- > > > Stefan Kupstaitis-Dunkler > > > > > > https://datahovel.com/ > > > (https://link.getmailspring.com/link/[email protected]/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D) > > > https://twitter.com/StefanDunkler > > > (https://link.getmailspring.com/link/[email protected]/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D) > > > > > > > > > > > > > > > > > > Virus-free. www.avg.com > > (http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient) > > -- > Stefan Kupstaitis-Dunkler > > https://datahovel.com/ > (https://link.getmailspring.com/link/[email protected]/0?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D) > https://twitter.com/StefanDunkler > (https://link.getmailspring.com/link/[email protected]/1?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D) > > > >
--- This email has been checked for viruses by AVG. https://www.avg.com
