Streaming Ingestion
Project: http://git-wip-us.apache.org/repos/asf/carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/carbondata/commit/b73a221a Tree: http://git-wip-us.apache.org/repos/asf/carbondata/tree/b73a221a Diff: http://git-wip-us.apache.org/repos/asf/carbondata/diff/b73a221a Branch: refs/heads/carbonstore Commit: b73a221a16fcbf0b14ad492e3ffc17902b15d4cd Parents: ab76347 Author: sgururajshetty <[email protected]> Authored: Mon Jan 29 19:34:14 2018 +0530 Committer: sgururajshetty <[email protected]> Committed: Mon Jan 29 19:34:14 2018 +0530 ---------------------------------------------------------------------- docs/configuration-parameters.md | 2 ++ docs/data-management-on-carbondata.md | 8 ++++++++ 2 files changed, 10 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/carbondata/blob/b73a221a/docs/configuration-parameters.md ---------------------------------------------------------------------- diff --git a/docs/configuration-parameters.md b/docs/configuration-parameters.md index fc10646..5794f4d 100644 --- a/docs/configuration-parameters.md +++ b/docs/configuration-parameters.md @@ -36,6 +36,8 @@ This section provides the details of all the configurations required for the Car | carbon.ddl.base.hdfs.url | hdfs://hacluster/opt/data | This property is used to configure the HDFS relative path, the path configured in carbon.ddl.base.hdfs.url will be appended to the HDFS path configured in fs.defaultFS. If this path is configured, then user need not pass the complete path while dataload. For example: If absolute path of the csv file is hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv, the path "hdfs://10.18.101.155:54310" will come from property fs.defaultFS and user can configure the /data/cnbc/ as carbon.ddl.base.hdfs.url. Now while dataload user can specify the csv path as /2016/xyz.csv. | | carbon.badRecords.location | /opt/Carbon/Spark/badrecords | Path where the bad records are stored. | | carbon.data.file.version | 2 | If this parameter value is set to 1, then CarbonData will support the data load which is in old format(0.x version). If the value is set to 2(1.x onwards version), then CarbonData will support the data load of new format only.| +| carbon.streaming.auto.handoff.enabled | true | If this parameter value is set to true, auto trigger handoff function will be enabled.| +| carbon.streaming.segment.max.size | 1024000000 | This parameter defines the maximum size of the streaming segment. Setting this parameter to appropriate value will avoid impacting the streaming ingestion. The value is in bytes.| ## Performance Configuration This section provides the details of all the configurations required for CarbonData Performance Optimization. http://git-wip-us.apache.org/repos/asf/carbondata/blob/b73a221a/docs/data-management-on-carbondata.md ---------------------------------------------------------------------- diff --git a/docs/data-management-on-carbondata.md b/docs/data-management-on-carbondata.md index a4fb40b..d1415ff 100644 --- a/docs/data-management-on-carbondata.md +++ b/docs/data-management-on-carbondata.md @@ -110,7 +110,14 @@ This tutorial is going to introduce all commands and data operations on CarbonDa 'COMPACTION_PRESERVE_SEGMENTS'='10', 'ALLOWED_COMPACTION_DAYS'='5') ``` + + - **Streaming** + + CarbonData supports streaming ingestion for real-time data. You can create the âstreamingâ table using the following table properties. + ``` + TBLPROPERTIES (âstreamingâ='trueâ) + ``` ### Example: @@ -134,6 +141,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa 'AUTO_LOAD_MERGE'='true', 'COMPACTION_LEVEL_THRESHOLD'='5,6', 'COMPACTION_PRESERVE_SEGMENTS'='10', + âstreamingâ='trueâ, 'ALLOWED_COMPACTION_DAYS'='5') ```
