Repository: carbondata-site Updated Branches: refs/heads/asf-site 0e4aec4a1 -> 324588f48
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/configuration-parameters.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/configuration-parameters.md b/src/site/markdown/configuration-parameters.md index 11cc6ea..46b8bd0 100644 --- a/src/site/markdown/configuration-parameters.md +++ b/src/site/markdown/configuration-parameters.md @@ -40,6 +40,8 @@ This section provides the details of all the configurations required for the Car | carbon.streaming.segment.max.size | 1024000000 | This parameter defines the maximum size of the streaming segment. Setting this parameter to appropriate value will avoid impacting the streaming ingestion. The value is in bytes.| | carbon.query.show.datamaps | true | If this parameter value is set to true, show tables command will list all the tables including datatmaps(eg: Preaggregate table), else datamaps will be excluded from the table list. | | carbon.segment.lock.files.preserve.hours | 48 | This property value indicates the number of hours the segment lock files will be preserved after dataload. These lock files will be deleted with the clean command after the configured number of hours. | +| carbon.unsafe.working.memory.in.mb | 512 | Specifies the size of executor unsafe working memory. Used for sorting data, storing column pages,etc. This value is expressed in MB. | +| carbon.unsafe.driver.working.memory.in.mb | 512 | Specifies the size of driver unsafe working memory. Used for storing block or blocklet datamap cache. If not configured then carbon.unsafe.working.memory.in.mb value is considered. This value is expressed in MB. | ## Performance Configuration This section provides the details of all the configurations required for CarbonData Performance Optimization. @@ -55,7 +57,11 @@ This section provides the details of all the configurations required for CarbonD | carbon.max.driver.lru.cache.size | -1 | Max LRU cache size upto which data will be loaded at the driver side. This value is expressed in MB. Default value of -1 means there is no memory limit for caching. Only integer values greater than 0 are accepted. | | | carbon.max.executor.lru.cache.size | -1 | Max LRU cache size upto which data will be loaded at the executor side. This value is expressed in MB. Default value of -1 means there is no memory limit for caching. Only integer values greater than 0 are accepted. If this parameter is not configured, then the carbon.max.driver.lru.cache.size value will be considered. | | | carbon.merge.sort.prefetch | true | Enable prefetch of data during merge sort while reading data from sort temp files in data loading. | | +| carbon.insert.persist.enable | false | Enabling this parameter considers persistent data. If we are executing insert into query from source table using select statement & loading the same source table concurrently, when select happens on source table during the data load, it gets new record for which dictionary is not generated, so there will be inconsistency. To avoid this condition we can persist the dataframe into MEMORY_AND_DISK(default value) and perform insert into operation. By default this value will be false because no need to persist the dataframe in all cases. If user wants to run load and insert queries on source table concurrently then user can enable this parameter. | | +| carbon.insert.storage.level | MEMORY_AND_DISK | Which storage level to persist dataframe when 'carbon.insert.persist.enable'=true, if user's executor has less memory, set this parameter to 'MEMORY_AND_DISK_SER' or other storage level to correspond to different environment. [See detail](http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence). | | | carbon.update.persist.enable | true | Enabling this parameter considers persistent data. Enabling this will reduce the execution time of UPDATE operation. | | +| carbon.update.storage.level | MEMORY_AND_DISK | Which storage level to persist dataframe when 'carbon.update.persist.enable'=true, if user's executor has less memory, set this parameter to 'MEMORY_AND_DISK_SER' or other storage level to correspond to different environment. [See detail](http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence). | | +| carbon.global.sort.rdd.storage.level | MEMORY_ONLY | Which storage level to persist rdd when loading data with 'sort_scope'='global_sort', if user's executor has less memory, set this parameter to 'MEMORY_AND_DISK_SER' or other storage level to correspond to different environment. [See detail](http://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence). | | | carbon.load.global.sort.partitions | 0 | The Number of partitions to use when shuffling data for sort. If user don't configurate or configurate it less than 1, it uses the number of map tasks as reduce tasks. In general, we recommend 2-3 tasks per CPU core in your cluster. | carbon.options.bad.records.logger.enable | false | Whether to create logs with details about bad records. | | | carbon.bad.records.action | FORCE | This property can have four types of actions for bad records FORCE, REDIRECT, IGNORE and FAIL. If set to FORCE then it auto-corrects the data by storing the bad records as NULL. If set to REDIRECT then bad records are written to the raw CSV instead of being loaded. If set to IGNORE then bad records are neither loaded nor written to the raw CSV. If set to FAIL then data loading fails if any bad records are found. | | @@ -63,7 +69,8 @@ This section provides the details of all the configurations required for CarbonD | carbon.options.bad.record.path | | Specifies the HDFS path where bad records are stored. By default the value is Null. This path must to be configured by the user if bad record logger is enabled or bad record action redirect. | | | carbon.enable.vector.reader | true | This parameter increases the performance of select queries as it fetch columnar batch of size 4*1024 rows instead of fetching data row by row. | | | carbon.blockletgroup.size.in.mb | 64 MB | The data are read as a group of blocklets which are called blocklet groups. This parameter specifies the size of the blocklet group. Higher value results in better sequential IO access.The minimum value is 16MB, any value lesser than 16MB will reset to the default value (64MB). | | -| carbon.task.distribution | block | **block**: Setting this value will launch one task per block. This setting is suggested in case of concurrent queries and queries having big shuffling scenarios. **custom**: Setting this value will group the blocks and distribute it uniformly to the available resources in the cluster. This enhances the query performance but not suggested in case of concurrent queries and queries having big shuffling scenarios. **blocklet**: Setting this value will launch one task per blocklet. This setting is suggested in case of concurrent queries and queries having big shuffling scenarios. **merge_small_files**: Setting this value will merge all the small partitions to a size of (128 MB is the default value of "spark.sql.files.maxPartitionBytes",it is configurable) during querying. The small partitions are combined to a map task to reduce the number of read task. This enhances the performance. | | +| carbon.task.distribution | block | **block**: Setting this value will launch one task per block. This setting is suggested in case of concurrent queries and queries having big shuffling scenarios. **custom**: Setting this value will group the blocks and distribute it uniformly to the available resources in the cluster. This enhances the query performance but not suggested in case of concurrent queries and queries having big shuffling scenarios. **blocklet**: Setting this value will launch one task per blocklet. This setting is suggested in case of concurrent queries and queries having big shuffling scenarios. **merge_small_files**: Setting this value will merge all the small partitions to a size of (128 MB is the default value of "spark.sql.files.maxPartitionBytes",it is configurable) during querying. The small partitions are combined to a map task to reduce the number of read task. This enhances the performance. | | +| carbon.load.sortmemory.spill.percentage | 0 | If we use unsafe memory during data loading, this configuration will be used to control the behavior of spilling inmemory pages to disk. Internally in Carbondata, during sorting carbondata will sort data in pages and add them in unsafe memory. If the memory is insufficient, carbondata will spill the pages to disk and generate sort temp file. This configuration controls how many pages in memory will be spilled to disk based size. The size can be calculated by multiplying this configuration value with 'carbon.sort.storage.inmemory.size.inmb'. For example, default value 0 means that no pages in unsafe memory will be spilled and all the newly sorted data will be spilled to disk; Value 50 means that if the unsafe memory is insufficient, about half of pages in the unsafe memory will be spilled to disk while value 100 means that almost all pages in unsafe memory will be spilled. **Note**: This configuration only works for 'LOCAL_SORT' and 'BA TCH_SORT' and the actual spilling behavior may slightly be different in each data loading. | Integer values between 0 and 100 | * **Compaction Configuration** @@ -76,7 +83,7 @@ This section provides the details of all the configurations required for CarbonD | carbon.horizontal.UPDATE.compaction.threshold | 1 | This property specifies the threshold limit on number of UPDATE delta files within a segment. In case the number of delta files goes beyond the threshold, the UPDATE delta files within the segment becomes eligible for horizontal compaction and compacted into single UPDATE delta file. | Values between 1 to 10000. | | carbon.horizontal.DELETE.compaction.threshold | 1 | This property specifies the threshold limit on number of DELETE delta files within a block of a segment. In case the number of delta files goes beyond the threshold, the DELETE delta files for the particular block of the segment becomes eligible for horizontal compaction and compacted into single DELETE delta file. | Values between 1 to 10000. | | carbon.update.segment.parallelism | 1 | This property specifies the parallelism for each segment during update. If there are segments that contain too many records to update and the spark job encounter data-spill related errors, it is better to increase this property value. It is recommended to set this value to a multiple of the number of executors for balance. | Values between 1 to 1000. | - +| carbon.merge.index.in.segment | true | This property is used to merge all carbon index files (.carbonindex) inside a segment to a single carbon index merge file (.carbonindexmerge).| Values true or false | * **Query Configuration** @@ -102,6 +109,12 @@ This section provides the details of all the configurations required for CarbonD |---------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | carbon.sort.file.write.buffer.size | 16384 | File write buffer size used during sorting. Minimum allowed buffer size is 10240 byte and Maximum allowed buffer size is 10485760 byte. | | carbon.lock.type | LOCALLOCK | This configuration specifies the type of lock to be acquired during concurrent operations on table. There are following types of lock implementation: - LOCALLOCK: Lock is created on local file system as file. This lock is useful when only one spark driver (thrift server) runs on a machine and no other CarbonData spark application is launched concurrently. - HDFSLOCK: Lock is created on HDFS file system as file. This lock is useful when multiple CarbonData spark applications are launched and no ZooKeeper is running on cluster and HDFS supports file based locking. | +| carbon.lock.path | TABLEPATH | Locks on the files are used to prevent concurrent operation from modifying the same files. This +configuration specifies the path where lock files have to be created. Recommended to configure +HDFS lock path(to this property) in case of S3 file system as locking is not feasible on S3. +**Note:** If this property is not set to HDFS location for S3 store, then there is a possibility +of data corruption because multiple data manipulation calls might try to update the status file +and as lock is not acquired before updation data might get overwritten. | | carbon.sort.intermediate.files.limit | 20 | Minimum number of intermediate files after which merged sort can be started (minValue = 2, maxValue=50). | | carbon.block.meta.size.reserved.percentage | 10 | Space reserved in percentage for writing block meta data in CarbonData file. | | carbon.csv.read.buffersize.byte | 1048576 | csv reading buffer size. | @@ -133,7 +146,7 @@ This section provides the details of all the configurations required for CarbonD | carbon.enableMinMax | true | Min max is feature added to enhance query performance. To disable this feature, set it false. | | carbon.dynamicallocation.schedulertimeout | 5 | Specifies the maximum time (unit in seconds) the scheduler can wait for executor to be active. Minimum value is 5 sec and maximum value is 15 sec. | | carbon.scheduler.minregisteredresourcesratio | 0.8 | Specifies the minimum resource (executor) ratio needed for starting the block distribution. The default value is 0.8, which indicates 80% of the requested resource is allocated for starting block distribution. The minimum value is 0.1 min and the maximum value is 1.0. | -| carbon.search.enabled | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. | +| carbon.search.enabled (Alpha Feature) | false | If set to true, it will use CarbonReader to do distributed scan directly instead of using compute framework like spark, thus avoiding limitation of compute framework like SQL optimizer and task scheduling overhead. | * **Global Dictionary Configurations** http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/data-management-on-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/data-management-on-carbondata.md b/src/site/markdown/data-management-on-carbondata.md index 706209c..0c54535 100644 --- a/src/site/markdown/data-management-on-carbondata.md +++ b/src/site/markdown/data-management-on-carbondata.md @@ -52,6 +52,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ``` TBLPROPERTIES ('DICTIONARY_INCLUDE'='column1, column2') ``` + NOTE: Dictionary Include/Exclude for complex child columns is not supported. - **Inverted Index Configuration** @@ -75,6 +76,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa OR TBLPROPERTIES ('SORT_COLUMNS'='') ``` + NOTE: Sort_Columns for Complex datatype columns is not supported. - **Sort Scope Configuration** @@ -85,6 +87,25 @@ This tutorial is going to introduce all commands and data operations on CarbonDa * BATCH_SORT: It increases the load performance but decreases the query performance if identified blocks > parallelism. * GLOBAL_SORT: It increases the query performance, especially high concurrent point query. And if you care about loading resources isolation strictly, because the system uses the spark GroupBy to sort data, the resource can be controlled by spark. + + ### Example: + + ``` + CREATE TABLE IF NOT EXISTS productSchema.productSalesTable ( + productNumber INT, + productName STRING, + storeCity STRING, + storeProvince STRING, + productCategory STRING, + productBatch STRING, + saleQuantity INT, + revenue INT) + STORED BY 'carbondata' + TBLPROPERTIES ('SORT_COLUMNS'='productName,storeCity', + 'SORT_SCOPE'='NO_SORT') + ``` + + **NOTE:** CarbonData also supports "using carbondata". Find example code at [SparkSessionExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/SparkSessionExample.scala) in the CarbonData repo. - **Table Block Size Configuration** @@ -122,23 +143,219 @@ This tutorial is going to introduce all commands and data operations on CarbonDa TBLPROPERTIES ('streaming'='true') ``` + - **Local Dictionary Configuration** + + Columns for which dictionary is not generated needs more storage space and in turn more IO. Also since more data will have to be read during query, query performance also would suffer.Generating dictionary per blocklet for such columns would help in saving storage space and assist in improving query performance as carbondata is optimized for handling dictionary encoded columns more effectively.Generating dictionary internally per blocklet is termed as local dictionary. Please refer to [File structure of Carbondata](../file-structure-of-carbondata.md) for understanding about the file structure of carbondata and meaning of terms like blocklet. + + Local Dictionary helps in: + 1. Getting more compression. + 2. Filter queries and full scan queries will be faster as filter will be done on encoded data. + 3. Reducing the store size and memory footprint as only unique values will be stored as part of local dictionary and corresponding data will be stored as encoded data. + 4. Getting higher IO throughput. + + **NOTE:** + + * Following Data Types are Supported for Local Dictionary: + * STRING + * VARCHAR + * CHAR + + * Following Data Types are not Supported for Local Dictionary: + * SMALLINT + * INTEGER + * BIGINT + * DOUBLE + * DECIMAL + * TIMESTAMP + * DATE + * BOOLEAN + + * In case of multi-level complex dataType columns, primitive string/varchar/char columns are considered for local dictionary generation. + + Local dictionary will have to be enabled explicitly during create table or by enabling the system property 'carbon.local.dictionary.enable'. By default, Local Dictionary will be disabled for the carbondata table. + + Local Dictionary can be configured using the following properties during create table command: + + | Properties | Default value | Description | + | ---------- | ------------- | ----------- | + | LOCAL_DICTIONARY_ENABLE | false | Whether to enable local dictionary generation. **NOTE:** If this property is defined, it will override the value configured at system level by 'carbon.local.dictionary.enable' | + | LOCAL_DICTIONARY_THRESHOLD | 10000 | The maximum cardinality of a column upto which carbondata can try to generate local dictionary (maximum - 100000) | + | LOCAL_DICTIONARY_INCLUDE | string/varchar/char columns| Columns for which Local Dictionary has to be generated.**NOTE:** Those string/varchar/char columns which are added into DICTIONARY_INCLUDE option will not be considered for local dictionary generation.| + | LOCAL_DICTIONARY_EXCLUDE | none | Columns for which Local Dictionary need not be generated. | + + **Fallback behavior:** + + * When the cardinality of a column exceeds the threshold, it triggers a fallback and the generated dictionary will be reverted and data loading will be continued without dictionary encoding. + + **NOTE:** When fallback is triggered, the data loading performance will decrease as encoded data will be discarded and the actual data is written to the temporary sort files. + + **Points to be noted:** + + 1. Reduce Block size: + + Number of Blocks generated is less in case of Local Dictionary as compression ratio is high. This may reduce the number of tasks launched during query, resulting in degradation of query performance if the pruned blocks are less compared to the number of parallel tasks which can be run. So it is recommended to configure smaller block size which in turn generates more number of blocks. + + 2. All the page-level data for a blocklet needs to be maintained in memory until all the pages encoded for local dictionary is processed in order to handle fallback. Hence the memory required for local dictionary based table is more and this memory increase is proportional to number of columns. + ### Example: - + ``` - CREATE TABLE IF NOT EXISTS productSchema.productSalesTable ( - productNumber INT, - productName STRING, - storeCity STRING, - storeProvince STRING, - productCategory STRING, - productBatch STRING, - saleQuantity INT, - revenue INT) - STORED BY 'carbondata' - TBLPROPERTIES ('SORT_COLUMNS'='productName,storeCity', - 'SORT_SCOPE'='NO_SORT') + CREATE TABLE carbontable( + + column1 string, + + column2 string, + + column3 LONG ) + + STORED BY 'carbondata' + TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE'='true','LOCAL_DICTIONARY_THRESHOLD'='1000', + 'LOCAL_DICTIONARY_INCLUDE'='column1','LOCAL_DICTIONARY_EXCLUDE'='column2') ``` - **NOTE:** CarbonData also supports "using carbondata". Find example code at [SparkSessionExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/SparkSessionExample.scala) in the CarbonData repo. + + **NOTE:** + + * We recommend to use Local Dictionary when cardinality is high but is distributed across multiple loads + * On a large cluster, decoding data can become a bottleneck for global dictionary as there will be many remote reads. In this scenario, it is better to use Local Dictionary. + * When cardinality is less, but loads are repetitive, it is better to use global dictionary as local dictionary generates multiple dictionary files at blocklet level increasing redundancy. + + - **Caching Min/Max Value for Required Columns** + By default, CarbonData caches min and max values of all the columns in schema. As the load increases, the memory required to hold the min and max values increases considerably. This feature enables you to configure min and max values only for the required columns, resulting in optimized memory usage. + + Following are the valid values for COLUMN_META_CACHE: + * If you want no column min/max values to be cached in the driver. + + ``` + COLUMN_META_CACHE=ââ + ``` + + * If you want only col1 min/max values to be cached in the driver. + + ``` + COLUMN_META_CACHE=âcol1â + ``` + + * If you want min/max values to be cached in driver for all the specified columns. + + ``` + COLUMN_META_CACHE=âcol1,col2,col3,â¦â + ``` + + Columns to be cached can be specified either while creating table or after creation of the table. + During create table operation; specify the columns to be cached in table properties. + + Syntax: + + ``` + CREATE TABLE [dbName].tableName (col1 String, col2 String, col3 int,â¦) STORED BY âcarbondataâ TBLPROPERTIES (âCOLUMN_META_CACHEâ=âcol1,col2,â¦â) + ``` + + Example: + + ``` + CREATE TABLE employee (name String, city String, id int) STORED BY âcarbondataâ TBLPROPERTIES (âCOLUMN_META_CACHEâ=ânameâ) + ``` + + After creation of table or on already created tables use the alter table command to configure the columns to be cached. + + Syntax: + + ``` + ALTER TABLE [dbName].tableName SET TBLPROPERTIES (âCOLUMN_META_CACHEâ=âcol1,col2,â¦â) + ``` + + Example: + + ``` + ALTER TABLE employee SET TBLPROPERTIES (âCOLUMN_META_CACHEâ=âcityâ) + ``` + + - **Caching at Block or Blocklet Level** + + This feature allows you to maintain the cache at Block level, resulting in optimized usage of the memory. The memory consumption is high if the Blocklet level caching is maintained as a Block can have multiple Blocklet. + + Following are the valid values for CACHE_LEVEL: + + *Configuration for caching in driver at Block level (default value).* + + ``` + CACHE_LEVEL= âBLOCKâ + ``` + + *Configuration for caching in driver at Blocklet level.* + + ``` + CACHE_LEVEL= âBLOCKLETâ + ``` + + Cache level can be specified either while creating table or after creation of the table. + During create table operation specify the cache level in table properties. + + Syntax: + + ``` + CREATE TABLE [dbName].tableName (col1 String, col2 String, col3 int,â¦) STORED BY âcarbondataâ TBLPROPERTIES (âCACHE_LEVELâ=âBlockletâ) + ``` + + Example: + + ``` + CREATE TABLE employee (name String, city String, id int) STORED BY âcarbondataâ TBLPROPERTIES (âCACHE_LEVELâ=âBlockletâ) + ``` + + After creation of table or on already created tables use the alter table command to configure the cache level. + + Syntax: + + ``` + ALTER TABLE [dbName].tableName SET TBLPROPERTIES (âCACHE_LEVELâ=âBlockletâ) + ``` + + Example: + + ``` + ALTER TABLE employee SET TBLPROPERTIES (âCACHE_LEVELâ=âBlockletâ) + ``` + + - **Support Flat folder same as Hive/Parquet** + + This feature allows all carbondata and index files to keep directy under tablepath. Currently all carbondata/carbonindex files written under tablepath/Fact/Part0/Segment_NUM folder and it is not same as hive/parquet folder structure. This feature makes all files written will be directly under tablepath, it does not maintain any segment folder structure.This is useful for interoperability between the execution engines and plugin with other execution engines like hive or presto becomes easier. + + Following table property enables this feature and default value is false. + ``` + 'flat_folder'='true' + ``` + Example: + ``` + CREATE TABLE employee (name String, city String, id int) STORED BY âcarbondataâ TBLPROPERTIES ('flat_folder'='true') + ``` + + - **String longer than 32000 characters** + + In common scenarios, the length of string is less than 32000, + so carbondata stores the length of content using Short to reduce memory and space consumption. + To support string longer than 32000 characters, carbondata introduces a table property called `LONG_STRING_COLUMNS`. + For these columns, carbondata internally stores the length of content using Integer. + + You can specify the columns as 'long string column' using below tblProperties: + + ``` + // specify col1, col2 as long string columns + TBLPROPERTIES ('LONG_STRING_COLUMNS'='col1,col2') + ``` + + Besides, you can also use this property through DataFrame by + ``` + df.format("carbondata") + .option("tableName", "carbonTable") + .option("long_string_columns", "col1, col2") + .save() + ``` + + If you are using Carbon-SDK, you can specify the datatype of long string column as `varchar`. + You can refer to SDKwriterTestCase for example. + + **NOTE:** The LONG_STRING_COLUMNS can only be string/char/varchar columns and cannot be dictionary_include/sort_columns/complex columns. ## CREATE TABLE AS SELECT This function allows user to create a Carbon table from any of the Parquet/Hive/Carbon table. This is beneficial when the user wants to create Carbon table from any other Parquet/Hive table and use the Carbon query engine to query and achieve better query results for cases where Carbon is faster than other file formats. Also this feature can be used for backing up the data. @@ -216,7 +433,12 @@ This tutorial is going to introduce all commands and data operations on CarbonDa This can be SDK output. Refer [SDK Writer Guide](https://github.com/apache/carbondata/blob/master/docs/sdk-writer-guide.md). **Note:** - Dropping of the external table should not delete the files present in the location. + 1. Dropping of the external table should not delete the files present in the location. + 2. When external table is created on non-transactional table data, + external table will be registered with the schema of carbondata files. + If multiple files with different schema is present, exception will be thrown. + So, If table registered with one schema and files are of different schema, + suggest to drop the external table and create again to register table with new schema. ## CREATE DATABASE @@ -285,6 +507,12 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ``` ALTER TABLE carbon ADD COLUMNS (a1 INT, b1 STRING) TBLPROPERTIES('DEFAULT.VALUE.a1'='10') ``` + NOTE: Add Complex datatype columns is not supported. + +Users can specify which columns to include and exclude for local dictionary generation after adding new columns. These will be appended with the already existing local dictionary include and exclude columns of main table respectively. + ``` + ALTER TABLE carbon ADD COLUMNS (a1 STRING, b1 STRING) TBLPROPERTIES('LOCAL_DICTIONARY_INCLUDE'='a1','LOCAL_DICTIONARY_EXCLUDE'='b1') + ``` - **DROP COLUMNS** @@ -301,6 +529,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ALTER TABLE carbon DROP COLUMNS (c1,d1) ``` + NOTE: Drop Complex child column is not supported. - **CHANGE DATA TYPE** @@ -324,6 +553,37 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ``` ALTER TABLE test_db.carbon CHANGE a1 a1 DECIMAL(18,2) ``` +- **MERGE INDEX** + + This command is used to merge all the CarbonData index files (.carbonindex) inside a segment to a single CarbonData index merge file (.carbonindexmerge). This enhances the first query performance. + ``` + ALTER TABLE [db_name.]table_name COMPACT 'SEGMENT_INDEX' + ``` + + Examples: + ``` + ALTER TABLE test_db.carbon COMPACT 'SEGMENT_INDEX' + ``` + **NOTE:** + * Merge index is not supported on streaming table. + +- **SET and UNSET for Local Dictionary Properties** + + When set command is used, all the newly set properties will override the corresponding old properties if exists. + + Example to SET Local Dictionary Properties: + ``` + ALTER TABLE tablename SET TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE'='false','LOCAL_DICTIONARY_THRESHOLD'='1000','LOCAL_DICTIONARY_INCLUDE'='column1','LOCAL_DICTIONARY_EXCLUDE'='column2') + ``` + When Local Dictionary properties are unset, corresponding default values will be used for these properties. + + Example to UNSET Local Dictionary Properties: + ``` + ALTER TABLE tablename UNSET TBLPROPERTIES('LOCAL_DICTIONARY_ENABLE','LOCAL_DICTIONARY_THRESHOLD','LOCAL_DICTIONARY_INCLUDE','LOCAL_DICTIONARY_EXCLUDE') + ``` + + **NOTE:** For old tables, by default, local dictionary is disabled. If user wants local dictionary for these tables, user can enable/disable local dictionary for new data at their discretion. + This can be achieved by using the alter table set command. ### DROP TABLE @@ -348,6 +608,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ``` REFRESH TABLE dbcarbon.productSalesTable ``` + **NOTE:** * The new database name and the old database name should be same. * Before executing this command the old table schema and data should be copied into the new database location. @@ -545,8 +806,18 @@ This tutorial is going to introduce all commands and data operations on CarbonDa * If the FORCE option is used, then it auto-converts the data by storing the bad records as NULL before Loading data. * If the IGNORE option is used, then bad records are neither loaded nor written to the separate CSV file. * In loaded data, if all records are bad records, the BAD_RECORDS_ACTION is invalid and the load operation fails. - * The maximum number of characters per column is 32000. If there are more than 32000 characters in a column, data loading will fail. - + * The default maximum number of characters per column is 32000. If there are more than 32000 characters in a column, please refer to *String longer than 32000 characters* section. + * Since Bad Records Path can be specified in create, load and carbon properties. + Therefore, value specified in load will have the highest priority, and value specified in carbon properties will have the least priority. + + **Bad Records Path:** + + This property is used to specify the location where bad records would be written. + + ``` + TBLPROPERTIES('BAD_RECORDS_PATH'='/opt/badrecords'') + ``` + Example: ``` @@ -555,6 +826,16 @@ This tutorial is going to introduce all commands and data operations on CarbonDa 'BAD_RECORDS_ACTION'='REDIRECT','IS_EMPTY_DATA_BAD_RECORD'='false') ``` + - **GLOBAL_SORT_PARTITIONS:** If the SORT_SCOPE is defined as GLOBAL_SORT, then user can specify the number of partitions to use while shuffling data for sort using GLOBAL_SORT_PARTITIONS. If it is not configured, or configured less than 1, then it uses the number of map task as reduce task. It is recommended that each reduce task deal with 512MB-1GB data. + + ``` + OPTIONS('GLOBAL_SORT_PARTITIONS'='2') + ``` + + NOTE: + * GLOBAL_SORT_PARTITIONS should be Integer type, the range is [1,Integer.MaxValue]. + * It is only used when the SORT_SCOPE is GLOBAL_SORT. + ### INSERT DATA INTO CARBONDATA TABLE This command inserts data into a CarbonData table, it is defined as a combination of two queries Insert and Select query respectively. @@ -641,6 +922,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ``` UPDATE t3 SET (t3_country, t3_salary) = (SELECT t5_country, t5_salary FROM t5 FULL JOIN t3 u WHERE u.t3_id = t5_id and t5_id=6) WHERE t3_id >6 ``` + NOTE: Update Complex datatype columns is not supported. ### DELETE @@ -710,6 +992,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ``` ALTER TABLE table_name COMPACT 'CUSTOM' WHERE SEGMENT.ID IN (2,3,4) ``` + NOTE: Compaction is unsupported for table containing Complex columns. - **CLEAN SEGMENTS AFTER Compaction** @@ -750,6 +1033,7 @@ This tutorial is going to introduce all commands and data operations on CarbonDa PARTITIONED BY (productCategory STRING, productBatch STRING) STORED BY 'carbondata' ``` + NOTE: Hive partition is not supported on complex datatype columns. #### Load Data Using Static Partition http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/datamap-management.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/datamap-management.md b/src/site/markdown/datamap-management.md new file mode 100644 index 0000000..b5d1aaa --- /dev/null +++ b/src/site/markdown/datamap-management.md @@ -0,0 +1,140 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData DataMap Management + +## Overview + +DataMap can be created using following DDL + +``` + CREATE DATAMAP [IF NOT EXISTS] datamap_name + [ON TABLE main_table] + USING "datamap_provider" + [WITH DEFERRED REBUILD] + DMPROPERTIES ('key'='value', ...) + AS + SELECT statement +``` + +Currently, there are 5 DataMap implementations in CarbonData. + +| DataMap Provider | Description | DMPROPERTIES | Management | +| ---------------- | ---------------------------------------- | ---------------------------------------- | ---------------- | +| preaggregate | single table pre-aggregate table | No DMPROPERTY is required | Automatic | +| timeseries | time dimension rollup table | event_time, xx_granularity, please refer to [Timeseries DataMap](https://github.com/apache/carbondata/blob/master/docs/datamap/timeseries-datamap-guide.md) | Automatic | +| mv | multi-table pre-aggregate table | No DMPROPERTY is required | Manual | +| lucene | lucene indexing for text column | index_columns to specifying the index columns | Automatic | +| bloomfilter | bloom filter for high cardinality column, geospatial column | index_columns to specifying the index columns | Automatic | + +## DataMap Management + +There are two kinds of management semantic for DataMap. + +1. Automatic Refresh: Create datamap without `WITH DEFERRED REBUILD` in the statement, which is by default. +2. Manual Refresh: Create datamap with `WITH DEFERRED REBUILD` in the statement + +**CAUTION:** +Manual refresh currently only works fine for MV, it has some bugs with other types of datamap in Carbondata 1.4.1, so we block this option for them in this version. +If user create MV datamap without specifying `WITH DEFERRED REBUILD`, carbondata will give a warning and treat the datamap as deferred rebuild. + +### Automatic Refresh + +When user creates a datamap on the main table without using `WITH DEFERRED REBUILD` syntax, the datamap will be managed by system automatically. +For every data load to the main table, system will immediately triger a load to the datamap automatically. These two data loading (to main table and datamap) is executed in a transactional manner, meaning that it will be either both success or neither success. + +The data loading to datamap is incremental based on Segment concept, avoiding a expesive total rebuild. + +If user perform following command on the main table, system will return failure. (reject the operation) + +1. Data management command: `UPDATE/DELETE/DELETE SEGMENT`. +2. Schema management command: `ALTER TABLE DROP COLUMN`, `ALTER TABLE CHANGE DATATYPE`, + `ALTER TABLE RENAME`. Note that adding a new column is supported, and for dropping columns and + change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if + not, the operation is allowed, otherwise operation will be rejected by throwing exception. +3. Partition management command: `ALTER TABLE ADD/DROP PARTITION + +If user do want to perform above operations on the main table, user can first drop the datamap, perform the operation, and re-create the datamap again. + +If user drop the main table, the datamap will be dropped immediately too. + +We do recommend you to use this management for index datamap. + +### Manual Refresh + +When user creates a datamap specifying maunal refresh semantic, the datamap is created with status *disabled* and query will NOT use this datamap until user can issue REBUILD DATAMAP command to build the datamap. For every REBUILD DATAMAP command, system will trigger a full rebuild of the datamap. After rebuild is done, system will change datamap status to *enabled*, so that it can be used in query rewrite. + +For every new data loading, data update, delete, the related datamap will be made *disabled*, +which means that the following queries will not benefit from the datamap before it becomes *enabled* again. + +If the main table is dropped by user, the related datamap will be dropped immediately. + +**Note**: ++ If you are creating a datamap on external table, you need to do manual management of the datamap. ++ For index datamap such as BloomFilter datamap, there is no need to do manual refresh. + By default it is automatic refresh, + which means its data will get refreshed immediately after the datamap is created or the main table is loaded. + Manual refresh on this datamap will has no impact. + + + +## DataMap Catalog + +Currently, when user creates a datamap, system will store the datamap metadata in a configurable *system* folder in HDFS or S3. + +In this *system* folder, it contains: + +- DataMapSchema file. It is a json file containing schema for one datamap. Ses DataMapSchema class. If user creates 100 datamaps (on different tables), there will be 100 files in *system* folder. +- DataMapStatus file. Only one file, it is in json format, and each entry in the file represents for one datamap. Ses DataMapStatusDetail class + +There is a DataMapCatalog interface to retrieve schema of all datamap, it can be used in optimizer to get the metadata of datamap. + + + +## DataMap Related Commands + +### Explain + +How can user know whether datamap is used in the query? + +User can use EXPLAIN command to know, it will print out something like + +```text +== CarbonData Profiler == +Hit mv DataMap: datamap1 +Scan Table: default.datamap1_table ++- filter: ++- pruning by CG DataMap ++- all blocklets: 1 + skipped blocklets: 0 +``` + +### Show DataMap + +There is a SHOW DATAMAPS command, when this is issued, system will read all datamap from *system* folder and print all information on screen. The current information includes: + +- DataMapName +- DataMapProviderName like mv, preaggreagte, timeseries, etc +- Associated Table + +### Compaction on DataMap + +This feature applies for preaggregate datamap only + +Running Compaction command (`ALTER TABLE COMPACT`) on main table will **not automatically** compact the pre-aggregate tables created on the main table. User need to run Compaction command separately on each pre-aggregate table to compact them. + +Compaction is an optional operation for pre-aggregate table. If compaction is performed on main table but not performed on pre-aggregate table, all queries still can benefit from pre-aggregate tables. To further improve the query performance, compaction on pre-aggregate tables can be triggered to merge the segments and files in the pre-aggregate tables. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/lucene-datamap-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/lucene-datamap-guide.md b/src/site/markdown/lucene-datamap-guide.md index 5f7a2e4..06cd194 100644 --- a/src/site/markdown/lucene-datamap-guide.md +++ b/src/site/markdown/lucene-datamap-guide.md @@ -1,4 +1,21 @@ -# CarbonData Lucene DataMap (Alpha feature in 1.4.0) +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# CarbonData Lucene DataMap (Alpha Feature) * [DataMap Management](#datamap-management) * [Lucene Datamap](#lucene-datamap-introduction) http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/preaggregate-datamap-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/preaggregate-datamap-guide.md b/src/site/markdown/preaggregate-datamap-guide.md index ce7cbcc..ff4c28e 100644 --- a/src/site/markdown/preaggregate-datamap-guide.md +++ b/src/site/markdown/preaggregate-datamap-guide.md @@ -1,3 +1,20 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + # CarbonData Pre-aggregate DataMap * [Quick Example](#quick-example) @@ -244,6 +261,7 @@ is not supported: change datatype command, CarbonData will check whether it will impact the pre-aggregate table, if not, the operation is allowed, otherwise operation will be rejected by throwing exception. 3. Partition management command: `ALTER TABLE ADD/DROP PARTITION` +4. Complex Datatypes for preaggregate is not supported. However, there is still way to support these operations on main table, in current CarbonData release, user can do as following: http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/s3-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/s3-guide.md b/src/site/markdown/s3-guide.md new file mode 100644 index 0000000..2f4dfa9 --- /dev/null +++ b/src/site/markdown/s3-guide.md @@ -0,0 +1,91 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +#S3 Guide (Alpha Feature 1.4.1) + +Object storage is the recommended storage format in cloud as it can support storing large data +files. S3 APIs are widely used for accessing object stores. This can be +used to store or retrieve data on Amazon cloud, Huawei Cloud(OBS) or on any other object + stores conforming to S3 API. +Storing data in cloud is advantageous as there are no restrictions on the size of +data and the data can be accessed from anywhere at any time. +Carbondata can support any Object Storage that conforms to Amazon S3 API. +Carbondata relies on Hadoop provided S3 filesystem APIs to access Object stores. + +#Writing to Object Storage + +To store carbondata files onto Object Store, `carbon.storelocation` property will have +to be configured with Object Store path in CarbonProperties file. + +For example: +``` +carbon.storelocation=s3a://mybucket/carbonstore. +``` + +If the existing store location cannot be changed or only specific tables need to be stored +onto cloud object store, it can be done so by specifying the `location` option in the create +table DDL command. + +For example: + +``` +CREATE TABLE IF NOT EXISTS db1.table1(col1 string, col2 int) STORED AS carbondata LOCATION 's3a://mybucket/carbonstore' +``` + +For more details on create table, Refer [data-management-on-carbondata](./data-management-on-carbondata.md#create-table) + +#Authentication + +Authentication properties will have to be configured to store the carbondata files on to S3 location. + +Authentication properties can be set in any of the following ways: +1. Set authentication properties in core-site.xml, refer +[hadoop authentication document](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authentication_properties) + +2. Set authentication properties in spark-defaults.conf. + +Example +``` +spark.hadoop.fs.s3a.secret.key=123 +spark.hadoop.fs.s3a.access.key=456 +``` + +3. Pass authentication properties with spark-submit as configuration. + +Example: +``` +./bin/spark-submit --master yarn --conf spark.hadoop.fs.s3a.secret.key=123 --conf spark.hadoop.fs +.s3a.access.key=456 --class= +``` + +4. Set authentication properties to hadoop configuration object in sparkContext. + +Example: +``` +sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key", "123") +sparkSession.sparkContext.hadoopConfiguration.set("fs.s3a.access.key","456") +``` + +#Recommendations + +1. Object Storage like S3 does not support file leasing mechanism(supported by HDFS) that is +required to take locks which ensure consistency between concurrent operations therefore, it is +recommended to set the configurable lock path property([carbon.lock.path](https://github.com/apache/carbondata/blob/master/docs/configuration-parameters.md#miscellaneous-configuration)) + to a HDFS directory. +2. Concurrent data manipulation operations are not supported. Object stores follow eventual +consistency semantics, i.e., any put request might take some time to reflect when trying to list +.This behaviour causes not to ensure the data read is always consistent or latest. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/sdk-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/sdk-guide.md b/src/site/markdown/sdk-guide.md index 5dbb5ac..e592aa5 100644 --- a/src/site/markdown/sdk-guide.md +++ b/src/site/markdown/sdk-guide.md @@ -1,3 +1,20 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + # SDK Guide In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader. # SDK Writer @@ -112,6 +129,49 @@ public class TestSdkAvro { } ``` +### Example with Json format +```java +import java.io.IOException; + +import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException; +import org.apache.carbondata.core.metadata.datatype.DataTypes; +import org.apache.carbondata.core.util.CarbonProperties; +import org.apache.carbondata.sdk.file.CarbonWriter; +import org.apache.carbondata.sdk.file.CarbonWriterBuilder; +import org.apache.carbondata.sdk.file.Field; +import org.apache.carbondata.sdk.file.Schema; + +public class TestSdkJson { + + public static void main(String[] args) throws InvalidLoadOptionException { + testJsonSdkWriter(); + } + + public static void testJsonSdkWriter() throws InvalidLoadOptionException { + String path = "./target/testJsonSdkWriter"; + + Field[] fields = new Field[2]; + fields[0] = new Field("name", DataTypes.STRING); + fields[1] = new Field("age", DataTypes.INT); + + Schema CarbonSchema = new Schema(fields); + + CarbonWriterBuilder builder = CarbonWriter.builder().outputPath(path); + + // initialize json writer with carbon schema + CarbonWriter writer = builder.buildWriterForJsonInput(CarbonSchema); + // one row of json Data as String + String JsonRow = "{\"name\":\"abcd\", \"age\":10}"; + + int rows = 5; + for (int i = 0; i < rows; i++) { + writer.write(JsonRow); + } + writer.close(); + } +} +``` + ## Datatypes Mapping Each of SQL data types are mapped into data types of SDK. Following are the mapping: @@ -128,6 +188,21 @@ Each of SQL data types are mapped into data types of SDK. Following are the mapp | STRING | DataTypes.STRING | | DECIMAL | DataTypes.createDecimalType(precision, scale) | +**NOTE:** + Carbon Supports below logical types of AVRO. + a. Date + The date logical type represents a date within the calendar, with no reference to a particular time zone or time of day. + A date logical type annotates an Avro int, where the int stores the number of days from the unix epoch, 1 January 1970 (ISO calendar). + b. Timestamp (millisecond precision) + The timestamp-millis logical type represents an instant on the global timeline, independent of a particular time zone or calendar, with a precision of one millisecond. + A timestamp-millis logical type annotates an Avro long, where the long stores the number of milliseconds from the unix epoch, 1 January 1970 00:00:00.000 UTC. + c. Timestamp (microsecond precision) + The timestamp-micros logical type represents an instant on the global timeline, independent of a particular time zone or calendar, with a precision of one microsecond. + A timestamp-micros logical type annotates an Avro long, where the long stores the number of microseconds from the unix epoch, 1 January 1970 00:00:00.000000 UTC. + + Currently the values of logical types are not validated by carbon. + Expect that avro record passed by the user is already validated by avro record generator tools. + ## Run SQL on files directly Instead of creating table and query it, you can also query that file directly with SQL. @@ -195,6 +270,23 @@ public CarbonWriterBuilder withBlockletSize(int blockletSize); ``` /** + * @param enableLocalDictionary enable local dictionary , default is false + * @return updated CarbonWriterBuilder + */ +public CarbonWriterBuilder enableLocalDictionary(boolean enableLocalDictionary); +``` + +``` +/** + * @param localDictionaryThreshold is localDictionaryThreshold,default is 10000 + * @return updated CarbonWriterBuilder + */ +public CarbonWriterBuilder localDictionaryThreshold(int localDictionaryThreshold) ; +``` + + +``` +/** * sets the list of columns that needs to be in sorted order * @param sortColumns is a string array of columns that needs to be sorted. * If it is null or by default all dimensions are selected for sorting @@ -222,7 +314,7 @@ public CarbonWriterBuilder persistSchemaFile(boolean persist); * by default it is system time in nano seconds. * @return updated CarbonWriterBuilder */ -public CarbonWriterBuilder taskNo(String taskNo); +public CarbonWriterBuilder taskNo(long taskNo); ``` ``` @@ -265,7 +357,7 @@ public CarbonWriterBuilder withLoadOptions(Map<String, String> options); * @throws IOException * @throws InvalidLoadOptionException */ -public CarbonWriter buildWriterForCSVInput() throws IOException, InvalidLoadOptionException; +public CarbonWriter buildWriterForCSVInput(org.apache.carbondata.sdk.file.Schema schema) throws IOException, InvalidLoadOptionException; ``` ``` @@ -276,15 +368,28 @@ public CarbonWriter buildWriterForCSVInput() throws IOException, InvalidLoadOpti * @throws IOException * @throws InvalidLoadOptionException */ -public CarbonWriter buildWriterForAvroInput() throws IOException, InvalidLoadOptionException; +public CarbonWriter buildWriterForAvroInput(org.apache.avro.Schema schema) throws IOException, InvalidLoadOptionException; +``` + +``` +/** +* Build a {@link CarbonWriter}, which accepts Json object +* @param carbonSchema carbon Schema object +* @return JsonCarbonWriter +* @throws IOException +* @throws InvalidLoadOptionException +*/ +public JsonCarbonWriter buildWriterForJsonInput(Schema carbonSchema); ``` ### Class org.apache.carbondata.sdk.file.CarbonWriter ``` /** * Write an object to the file, the format of the object depends on the implementation -* If AvroCarbonWriter, object is of type org.apache.avro.generic.GenericData.Record -* If CSVCarbonWriter, object is of type String[] +* If AvroCarbonWriter, object is of type org.apache.avro.generic.GenericData.Record, +* which is one row of data. +* If CSVCarbonWriter, object is of type String[], which is one row of data +* If JsonCarbonWriter, object is of type String, which is one row of json * Note: This API is not thread safe * @param object * @throws IOException @@ -460,16 +565,6 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ ``` /** - * Project all Columns for carbon reader - * - * @return CarbonReaderBuilder object - * @throws IOException - */ - public CarbonReaderBuilder projectAllColumns(); -``` - -``` - /** * Configure the transactional status of table * If set to false, then reads the carbondata and carbonindex files from a flat folder structure. * If set to true, then reads the carbondata and carbonindex files from segment folder structure. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/streaming-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/streaming-guide.md b/src/site/markdown/streaming-guide.md index a9b174f..32d24dc 100644 --- a/src/site/markdown/streaming-guide.md +++ b/src/site/markdown/streaming-guide.md @@ -1,3 +1,20 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + # CarbonData Streaming Ingestion ## Quick example @@ -28,6 +45,7 @@ Start spark-shell in new terminal, type :paste, then copy and run the following import org.apache.spark.sql.CarbonSession._ import org.apache.spark.sql.streaming.{ProcessingTime, StreamingQuery} import org.apache.carbondata.core.util.path.CarbonTablePath + import org.apache.carbondata.streaming.parser.CarbonStreamParser val warehouse = new File("./warehouse").getCanonicalPath val metastore = new File("./metastore").getCanonicalPath @@ -71,6 +89,8 @@ Start spark-shell in new terminal, type :paste, then copy and run the following .option("checkpointLocation", CarbonTablePath.getStreamingCheckpointDir(tablePath)) .option("dbName", "default") .option("tableName", "carbon_table") + .option(CarbonStreamParser.CARBON_STREAM_PARSER, + CarbonStreamParser.CARBON_STREAM_PARSER_CSV) .start() // start new thread to show data @@ -157,13 +177,13 @@ Config the property "carbon.stream.parser" to define a stream parser to convert property name | default | description --- | --- | --- -carbon.stream.parser | org.apache.carbondata.streaming.parser.CSVStreamParserImp | the class of the stream parser +carbon.stream.parser | org.apache.carbondata.streaming.parser.RowStreamParserImp | the class of the stream parser Currently CarbonData support two parsers, as following: -**1. org.apache.carbondata.streaming.parser.CSVStreamParserImp**: This is the default stream parser, it gets a line data(String type) from the first index of InternalRow and converts this String to Object[]. +**1. org.apache.carbondata.streaming.parser.CSVStreamParserImp**: This parser gets a line data(String type) from the first index of InternalRow and converts this String to Object[]. -**2. org.apache.carbondata.streaming.parser.RowStreamParserImp**: This stream parser will auto convert InternalRow to Object[] according to schema of this `DataSet`, for example: +**2. org.apache.carbondata.streaming.parser.RowStreamParserImp**: This is the default stream parser, it will auto convert InternalRow to Object[] according to schema of this `DataSet`, for example: ```scala case class FileElement(school: Array[String], age: Int) @@ -191,8 +211,6 @@ Currently CarbonData support two parsers, as following: .option("checkpointLocation", tablePath.getStreamingCheckpointDir) .option("dbName", "default") .option("tableName", "carbon_table") - .option(CarbonStreamParser.CARBON_STREAM_PARSER, - CarbonStreamParser.CARBON_STREAM_PARSER_ROW_PARSER) .start() ... http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/supported-data-types-in-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/supported-data-types-in-carbondata.md b/src/site/markdown/supported-data-types-in-carbondata.md index 7260afe..eb74a2e 100644 --- a/src/site/markdown/supported-data-types-in-carbondata.md +++ b/src/site/markdown/supported-data-types-in-carbondata.md @@ -35,6 +35,9 @@ * CHAR * VARCHAR + **NOTE**: For string longer than 32000 characters, use `LONG_STRING_COLUMNS` in table property. + Please refer to TBLProperties in [CreateTable](https://github.com/apache/carbondata/blob/master/docs/data-management-on-carbondata.md#create-table) for more information. + * Complex Types * arrays: ARRAY``<data_type>`` * structs: STRUCT``<col_name : data_type COMMENT col_comment, ...>`` http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/timeseries-datamap-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/timeseries-datamap-guide.md b/src/site/markdown/timeseries-datamap-guide.md index bea5286..135188d 100644 --- a/src/site/markdown/timeseries-datamap-guide.md +++ b/src/site/markdown/timeseries-datamap-guide.md @@ -1,10 +1,27 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + # CarbonData Timeseries DataMap * [Timeseries DataMap Introduction](#timeseries-datamap-intoduction) * [Compaction](#compacting-pre-aggregate-tables) * [Data Management](#data-management-with-pre-aggregate-tables) -## Timeseries DataMap Introduction (Alpha feature in 1.3.0) +## Timeseries DataMap Introduction (Alpha Feature) Timeseries DataMap a pre-aggregate table implementation based on 'pre-aggregate' DataMap. Difference is that Timeseries DataMap has built-in understanding of time hierarchy and levels: year, month, day, hour, minute, so that it supports automatic roll-up in time dimension http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/markdown/useful-tips-on-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/useful-tips-on-carbondata.md b/src/site/markdown/useful-tips-on-carbondata.md index ff339d0..641a7f3 100644 --- a/src/site/markdown/useful-tips-on-carbondata.md +++ b/src/site/markdown/useful-tips-on-carbondata.md @@ -17,7 +17,7 @@ # Useful Tips This tutorial guides you to create CarbonData Tables and optimize performance. - The following sections will elaborate on the above topics : + The following sections will elaborate on the below topics : * [Suggestions to create CarbonData Table](#suggestions-to-create-carbondata-table) * [Configuration for Optimizing Data Loading performance for Massive Data](#configuration-for-optimizing-data-loading-performance-for-massive-data) @@ -125,6 +125,10 @@ TBLPROPERTIES ('SORT_COLUMNS'='Dime_1, HOST, MSISDN') ``` + **NOTE:** + + BloomFilter can be created to enhance performance for queries with precise equal/in conditions. You can find more information about it in BloomFilter datamap [document](https://github.com/apache/carbondata/blob/master/docs/datamap/bloomfilter-datamap-guide.md). + + ## Configuration for Optimizing Data Loading performance for Massive Data @@ -168,7 +172,8 @@ | carbon.detail.batch.size | spark/carbonlib/carbon.properties | Data loading | The buffer size to store records, returned from the block scan. | In limit scenario this parameter is very important. For example your query limit is 1000. But if we set this value to 3000 that means we get 3000 records from scan but spark will only take 1000 rows. So the 2000 remaining are useless. In one Finance test case after we set it to 100, in the limit 1000 scenario the performance increase about 2 times in comparison to if we set this value to 12000. | | carbon.use.local.dir | spark/carbonlib/carbon.properties | Data loading | Whether use YARN local directories for multi-table load disk load balance | If this is set it to true CarbonData will use YARN local directories for multi-table load disk load balance, that will improve the data load performance. | | carbon.use.multiple.temp.dir | spark/carbonlib/carbon.properties | Data loading | Whether to use multiple YARN local directories during table data loading for disk load balance | After enabling 'carbon.use.local.dir', if this is set to true, CarbonData will use all YARN local directories during data load for disk load balance, that will improve the data load performance. Please enable this property when you encounter disk hotspot problem during data loading. | - | carbon.sort.temp.compressor | spark/carbonlib/carbon.properties | Data loading | Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading. | The optional values are 'SNAPPY','GZIP','BZIP2','LZ4' and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck. | + | carbon.sort.temp.compressor | spark/carbonlib/carbon.properties | Data loading | Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading. | The optional values are 'SNAPPY','GZIP','BZIP2','LZ4','ZSTD' and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck. | | carbon.load.skewedDataOptimization.enabled | spark/carbonlib/carbon.properties | Data loading | Whether to enable size based block allocation strategy for data loading. | When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB~1GB. | - + | carbon.load.min.size.enabled | spark/carbonlib/carbon.properties | Data loading | Whether to enable node minumun input data size allocation strategy for data loading.| When loading, carbondata will use node minumun input data size allocation strategy for task distribution. It will make sure the node load the minimum amount of data -- It's useful if the size of your input data files very small, say 1MB~256MB,Avoid generating a large number of small files. | + Note: If your CarbonData instance is provided only for query, you may specify the property 'spark.speculation=true' which is in conf directory of spark. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/14d670df/src/site/pdf.xml ---------------------------------------------------------------------- diff --git a/src/site/pdf.xml b/src/site/pdf.xml index 5fc3341..0f83c68 100644 --- a/src/site/pdf.xml +++ b/src/site/pdf.xml @@ -17,9 +17,11 @@ <item name="Configuring CarbonData" ref='configuration-parameters.md'/> <item name="Streaming Guide" ref='streaming-guide.md'/> <item name="SDK Guide" ref='sdk-guide.md'/> + <item name="S3 Guide (Alpha Feature)" ref='s3-guide.md'/> <item name="DataMap Developer Guide" ref='datamap-developer-guide.md'/> - <item name="CarbonData BloomFilter DataMap (Alpha feature in 1.4.0)" ref='bloomfilter-datamap-guide.md'/> - <item name="CarbonData Lucene DataMap (Alpha feature in 1.4.0)" ref='lucene-datamap-guide.md'/> + <item name="CarbonData DataMap Management" ref='datamap-management.md'/> + <item name="CarbonData BloomFilter DataMap (Alpha Feature)" ref='bloomfilter-datamap-guide.md'/> + <item name="CarbonData Lucene DataMap (Alpha Feature)" ref='lucene-datamap-guide.md'/> <item name="CarbonData Pre-aggregate DataMap" ref='preaggregate-datamap-guide.md'/> <item name="CarbonData Timeseries DataMap" ref='timeseries-datamap-guide.md'/> <item name="FAQs" ref='faq.md'/> @@ -32,7 +34,7 @@ <companyLogo>../../src/site/projectLogo/ApacheLogo.png</companyLogo> <projectLogo>../../src/site/projectLogo/CarbonDataLogo.png</projectLogo> <coverTitle>Apache CarbonData</coverTitle> - <coverSubTitle>Ver 1.4.0 </coverSubTitle> + <coverSubTitle>Ver 1.4.1 </coverSubTitle> <coverType>Documentation</coverType> <projectName>Apache CarbonData</projectName> <companyName>The Apache Software Foundation</companyName>
