Re: Camel + Drill + Parquet

2020-02-13 Thread Omar Al-Safi
No worries Ron! And enjoy the ride :) Regards, Omar On Thu, Feb 13, 2020 at 6:39 AM Ron Cecchini wrote: > Thanks, Omar. > > As it turns out, Parquet is not the way to go since it looks like it is > geared more toward data warehousing, whereas I need to persist streaming > data - and from what

Re: Camel + Drill + Parquet

2020-02-12 Thread Ron Cecchini
Thanks, Omar. As it turns out, Parquet is not the way to go since it looks like it is geared more toward data warehousing, whereas I need to persist streaming data - and from what I can gather, I would need the overhead of Spark or Hive to accomplish that with Parquet (appending to a growing

Re: Camel + Drill + Parquet

2020-02-12 Thread Omar Al-Safi
Hi Ron, By reading some introduction in Apache Drill, I'd say the file component would be more suitable to write parquet files. In regards to parquet and Camel, we don't have an example for it but the way I see it, you are heading into the right direction by creating a processor to convert the