I know this is not an "out of the box" solution, but you could write
something quick outside of Metron that consumes from the 'indexing' topic,
and lands each message in a dedicated topic based on the 'source.type'.  I
have never used Kafka Connect, so don't have any insights there.

Unfortunately, we do not make it easy right now to alter the standard data
flow in Metron.  I think that is something that we could improve upon
though.

On Mon, Sep 19, 2016 at 9:48 AM, Mao, Kevin <kevin....@capitalone.com>
wrote:

> Our primary use case is that we want to ingest enriched data into Amazon
> S3 using Kafka Connect, and have it formatted as ORC/Parquet files.
>
>
>
> Having separate topics for each type of enriched data would allow us to
> serialize each topic using individual Avro schemas. From there, we’d be
> able to automatically deserialize and convert each type of data into their
> own datasets for later batch processing (e.g. Hive tables backed by ORC
> files).
>
>
>
> If there’s a more optimal way to get enriched data into S3, I’m certainly
> open to suggestions.
>
>
>
> Thanks!
>
>
>
> - Kevin
>
>
>
>
>
> *From: *Nick Allen <n...@nickallen.org>
> *Reply-To: *"user@metron.incubator.apache.org" <
> user@metron.incubator.apache.org>
> *Date: *Saturday, September 17, 2016 at 10:01 AM
> *To: *"user@metron.incubator.apache.org" <user@metron.incubator.apache.org
> >
> *Subject: *Re: Hi
>
>
>
> The enriched data has a 'source.type' field that you can use to
> distinguish the original source of enriched data.  Do separate topics buy
> you anything else?
>
>
>
>
>
>
>
>
>
> On Thu, Sep 15, 2016 at 11:55 AM, Mao, Kevin <kevin....@capitalone.com>
> wrote:
>
> Hi James,
>
> One feature that I would find useful would be optional support for writing
> enriched data to separate Kafka topics for each sensor type (e.g.
> “bluecoatcim-enriched”, “paloalto-enriched”, etc.). This would put us on
> the road to getting the data into a more well-structured format for later
> batch analysis. WDYT?
>
> - Kevin
>
>
>
> On 9/15/16, 12:27 AM, "James Sirota" <jsir...@apache.org> wrote:
>
>     Hi Kevin, welcome to the community.  What features would you like to
> work on?
>
>     14.09.2016, 13:55, "Mao, Kevin" <kevin....@capitalone.com>:
>     > Hello,
>     >
>     > My name is Kevin Mao and I’m a data engineer with 5 years of
> experience working on the Data Ingestion team at CapitalOne. I’m looking
> forward to working with you guys!
>     >
>     > Kevin
>     >
>     > ----------------------------------------
>     >
>     > The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>     -------------------
>     Thank you,
>
>     James Sirota
>     PPMC- Apache Metron (Incubating)
>     jsirota AT apache DOT org
>
> ________________________________________________________
>
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>
>
>
>
> --
>
> Nick Allen <n...@nickallen.org>
>
> ------------------------------
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>



-- 
Nick Allen <n...@nickallen.org>

Reply via email to