[
https://issues.apache.org/jira/browse/CARBONDATA-1572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205412#comment-16205412
]
Jacky Li edited comment on CARBONDATA-1572 at 10/16/17 4:05 AM:
----------------------------------------------------------------
My understanding is:
1. Handoff process is like compaction process, it will first read files in
streaming segment and create new columnar files, after all files are converted,
mark the old streaming segment for deletion. It will be delete after timeout or
user can delete it manually by CLEAN command.
2. As a first step, I think landing on HDFS directly is simpler and it can
provide data consistency also. The only problem is the write throughput may not
be comparable to HBase. But if we implement like HBase, we need many
infrastructures like mem-store, WAL, recovery process, etc. It is long way to
go.
was (Author: jackylk):
My understanding is:
1. Handoff process is like compaction process, it will first read files in
streaming segment and create new columnar files, after all files are converted,
mark the old streaming segment for deletion. It will be delete after timeout or
user can delete it manually by CLEAN command.
2. As a first step, I think landing on HDFS directly is simpler. If we
implement like HBase, we need many infrastructures like mem-store, WAL,
recovery process, etc. It is long way to go.
> Support Streaming Ingest
> ------------------------
>
> Key: CARBONDATA-1572
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1572
> Project: CarbonData
> Issue Type: New Feature
> Reporter: QiangCai
> Attachments: CarbonData Streaming Ingest.pdf
>
>
> CarbonData should support streaming ingest.
> [^CarbonData Streaming Ingest.pdf]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
