Hi Nathanael, I would like to contribute with this implementation, or if there is any task related. I can help Elio Salvatore.
Regards. On Tue, Feb 27, 2018 at 11:04 AM, Nate Smith <[email protected]> wrote: > Hello, > > This email seemed to have slipped through the cracks. > How is development going? > Is this still something you are interested in contributing? > > - nathanael > > > On Sep 15, 2017, at 5:17 AM, Salvatore Elio <[email protected]> > wrote: > > > > Hello, > > > > > > > > for an internal project we have developed a different ingestion process > for DNS in order to have a real time ingestion and support early enrichment > of the ingested data. > > > > The ingestion process is splitted into 2 processes: > > > > > > > > 1) From DNS data to Kafka - An Akka Streams job based on Pcap4j > (https://<https://github.com/kaitoy/pcap4j)>github.com/kaitoy/pcap4j< > https://github.com/kaitoy/pcap4j)>) which: > > > > a. loop through all the filtered UDP packets on port 53 using Pcap4j; > > > > b. convert Pcap4j packet objects to AVRO using Twitter Bijection; > > > > c. send Avro objects to Kafka. > > > > > > > > 2) From Kafka to hive - a Spark Streaming job that read new messages on > Kafka and write them in partitioned HDFS parquet folder readable by > Hive/Impala. > > > > > > We would like to know your thoughts about this and if this could be > integrated into apache spot. If it is of interest we can share the code. > > > > > > > > > > > > Thanks > > > > [SIA logo] > > ________________________________ > > > > *******************Internet Email Confidentiality > Footer******************* > > Qualsiasi utilizzo non autorizzato del presente messaggio nonché dei > suoi allegati è vietato e potrebbe costituire reato. Se ha ricevuto per > errore il presente messaggio, Le saremmo grati se ci inviasse, via e-mail, > una comunicazione al riguardo e provvedesse nel contempo alla distruzione > del messaggio stesso e dei suoi eventuali allegati. Le dichiarazioni > contenute nel presente messaggio nonche' nei suoi eventuali allegati devono > essere attribuite al mittente e non possono essere necessariamente > considerate come autorizzate da SIA S.p.A.; le medesime dichiarazioni non > impegnano SIA S.p.A. nei confronti del destinatario o di terzi. SIA S.p.A. > non si assume alcuna responsabilita' per eventuali intercettazioni, > modifiche o danneggiamenti del presente messaggio e-mail. > > > > Any unauthorized use of this e-mail or any of its attachments is > prohibited and could constitute an offence. If you are not the intended > addressee please advise immediately the sender by using the reply facility > in your e-mail software and destroy the message and its attachments. The > statements and opinions expressed in this e-mail message are those of the > author of the message and do not necessarily represent those of SIA S.p.A. > Besides, The contents of this message shall be understood as neither given > nor endorsed by SIA S.p.A.. SIA S.p.A. does not accept liability for > corruption, interception or amendment, if any, or the consequences thereof. >
