Hi,
there the right answer depends on (at least) two aspects:
a) Do you have an actual streaming case or is it batch, i.e., does the data
come from a potentially infinite stream or not? This basically determines
the system to handle your data.
- Stream: I don't have much experience here, but Flink's new
Streaming feature, Kafka or Flume might be worth looking at.
- Batch: A regular Flink job might work.
b) How do you want to access your data? This influences the format to store
the data.
- Full scans of some columns (large fraction of tuples) -> Parquet or
ORC in HDFS
- Point access to certain tuples (also subsets of columns, few or
many tuples) -> HBase,
- always read all full tuples -> Avro, ProtoBufs in HDFS
I don't know how much throughput these systems are able to handle though...
Hope this helps,
Fabian
2014-09-29 17:32 GMT+02:00 Flavio Pompermaier <[email protected]>:
> Hi guys,
>
> in my use case I have burst of data coming into my system (RDF triples
> generated from a CSV that I need to process in a further step) and I was
> trying to figure it out what is the best way to save them on HDFS.
> Do you suggest me to save them on HBase or to use a serialization tool
> like avro/parquet and similar? Do I need Flume as well or there's a Flink
> solution for that?
>
> Best,
> Flavio
>
