[
https://issues.apache.org/jira/browse/CARBONDATA-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16712447#comment-16712447
]
Nicholas Jiang commented on CARBONDATA-3130:
--------------------------------------------
# Each flink process, use the SDK to create a directory of its own online
segment (use UUID for the directory name)
# Then continuously write new data files to this directory. There is also an
index file in this directory to record which data files are valid.
# When the online segment directory reaches a certain size, the handoff action
is triggered, that is, the table status metadata is modified. The SDK is then
responsible for creating a new online segment directory. Then repeat step 2.
# For the online segment query, first read the index file to get a list of
valid data files, and then read each file.
* The role of the index file is to avoid reading half of the flush data file
when querying.In the index file, you need to record a list of valid data files,
or you can add some minmax statistics.When reading, first read the index file,
get the data file path, list of path, and then read these files.The name of the
valid data file in the current online segment directory is written in the index
file.
* Each process has an online segment directory to ensure that each process can
write concurrently. This mechanism can be used in scenarios without a central
collaborator, such as flink, kafka stream, cassandra, etc.
* Reading while writing, refers to one side using flink into the library, while
using spark/presto query, in this case can not let the query side read to not
write a complete data file.
* The essential difference between online segment and stream segment is that
the former is the process level (no scheduling, multiple active), the latter is
application level (with scheduling, only one active).
> CarbonData Support Flink
> ------------------------
>
> Key: CARBONDATA-3130
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3130
> Project: CarbonData
> Issue Type: New Feature
> Components: flink-integration
> Reporter: Nicholas Jiang
> Assignee: Nicholas Jiang
> Priority: Minor
>
> For streaming warehousing scenarios，CarbonData support flink.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
