Our primary use case is that we want to ingest enriched data into Amazon S3 
using Kafka Connect, and have it formatted as ORC/Parquet files.

Having separate topics for each type of enriched data would allow us to 
serialize each topic using individual Avro schemas. From there, we’d be able to 
automatically deserialize and convert each type of data into their own datasets 
for later batch processing (e.g. Hive tables backed by ORC files).

If there’s a more optimal way to get enriched data into S3, I’m certainly open 
to suggestions.

Thanks!

- Kevin


From: Nick Allen <n...@nickallen.org>
Reply-To: "user@metron.incubator.apache.org" <user@metron.incubator.apache.org>
Date: Saturday, September 17, 2016 at 10:01 AM
To: "user@metron.incubator.apache.org" <user@metron.incubator.apache.org>
Subject: Re: Hi

The enriched data has a 'source.type' field that you can use to distinguish the 
original source of enriched data.  Do separate topics buy you anything else?




On Thu, Sep 15, 2016 at 11:55 AM, Mao, Kevin 
<kevin....@capitalone.com<mailto:kevin....@capitalone.com>> wrote:
Hi James,

One feature that I would find useful would be optional support for writing 
enriched data to separate Kafka topics for each sensor type (e.g. 
“bluecoatcim-enriched”, “paloalto-enriched”, etc.). This would put us on the 
road to getting the data into a more well-structured format for later batch 
analysis. WDYT?

- Kevin


On 9/15/16, 12:27 AM, "James Sirota" 
<jsir...@apache.org<mailto:jsir...@apache.org>> wrote:

    Hi Kevin, welcome to the community.  What features would you like to work 
on?

    14.09.2016, 13:55, "Mao, Kevin" 
<kevin....@capitalone.com<mailto:kevin....@capitalone.com>>:
    > Hello,
    >
    > My name is Kevin Mao and I’m a data engineer with 5 years of experience 
working on the Data Ingestion team at CapitalOne. I’m looking forward to 
working with you guys!
    >
    > Kevin
    >
    > ----------------------------------------
    >
    > The information contained in this e-mail is confidential and/or 
proprietary to Capital One and/or its affiliates and may only be used solely in 
performance of work or services for Capital One. The information transmitted 
herewith is intended only for use by the individual or entity to which it is 
addressed. If the reader of this message is not the intended recipient, you are 
hereby notified that any review, retransmission, dissemination, distribution, 
copying or other use of, or taking of any action in reliance upon this 
information is strictly prohibited. If you have received this communication in 
error, please contact the sender and delete the material from your computer.

    -------------------
    Thank you,

    James Sirota
    PPMC- Apache Metron (Incubating)
    jsirota AT apache DOT org

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.



--
Nick Allen <n...@nickallen.org<mailto:n...@nickallen.org>>
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Reply via email to