Thank you Nick I am using Kafka broker for my Big Data projects. Please share me the problem so that I will provide some solution.
Can we alter data flow of Metron using Kafka broker? if yes, please provide me insight I will transform into Kafka Best Regards, Ravi, On Tue, Sep 20, 2016 at 2:20 PM, Nick Allen <n...@nickallen.org> wrote: > I know this is not an "out of the box" solution, but you could write > something quick outside of Metron that consumes from the 'indexing' topic, > and lands each message in a dedicated topic based on the 'source.type'. I > have never used Kafka Connect, so don't have any insights there. > > Unfortunately, we do not make it easy right now to alter the standard data > flow in Metron. I think that is something that we could improve upon > though. > > On Mon, Sep 19, 2016 at 9:48 AM, Mao, Kevin <kevin....@capitalone.com> > wrote: > >> Our primary use case is that we want to ingest enriched data into Amazon >> S3 using Kafka Connect, and have it formatted as ORC/Parquet files. >> >> >> >> Having separate topics for each type of enriched data would allow us to >> serialize each topic using individual Avro schemas. From there, we’d be >> able to automatically deserialize and convert each type of data into their >> own datasets for later batch processing (e.g. Hive tables backed by ORC >> files). >> >> >> >> If there’s a more optimal way to get enriched data into S3, I’m certainly >> open to suggestions. >> >> >> >> Thanks! >> >> >> >> - Kevin >> >> >> >> >> >> *From: *Nick Allen <n...@nickallen.org> >> *Reply-To: *"user@metron.incubator.apache.org" < >> user@metron.incubator.apache.org> >> *Date: *Saturday, September 17, 2016 at 10:01 AM >> *To: *"user@metron.incubator.apache.org" <user@metron.incubator.apache. >> org> >> *Subject: *Re: Hi >> >> >> >> The enriched data has a 'source.type' field that you can use to >> distinguish the original source of enriched data. Do separate topics buy >> you anything else? >> >> >> >> >> >> >> >> >> >> On Thu, Sep 15, 2016 at 11:55 AM, Mao, Kevin <kevin....@capitalone.com> >> wrote: >> >> Hi James, >> >> One feature that I would find useful would be optional support for >> writing enriched data to separate Kafka topics for each sensor type (e.g. >> “bluecoatcim-enriched”, “paloalto-enriched”, etc.). This would put us on >> the road to getting the data into a more well-structured format for later >> batch analysis. WDYT? >> >> - Kevin >> >> >> >> On 9/15/16, 12:27 AM, "James Sirota" <jsir...@apache.org> wrote: >> >> Hi Kevin, welcome to the community. What features would you like to >> work on? >> >> 14.09.2016, 13:55, "Mao, Kevin" <kevin....@capitalone.com>: >> > Hello, >> > >> > My name is Kevin Mao and I’m a data engineer with 5 years of >> experience working on the Data Ingestion team at CapitalOne. I’m looking >> forward to working with you guys! >> > >> > Kevin >> > >> > ---------------------------------------- >> > >> > The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> >> ------------------- >> Thank you, >> >> James Sirota >> PPMC- Apache Metron (Incubating) >> jsirota AT apache DOT org >> >> ________________________________________________________ >> >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> >> >> >> >> >> -- >> >> Nick Allen <n...@nickallen.org> >> >> ------------------------------ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> > > > > -- > Nick Allen <n...@nickallen.org> >
