Repository: carbondata-site Updated Branches: refs/heads/asf-site 4574eccb4 -> ae77df2e4
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/ddl-of-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/ddl-of-carbondata.md b/src/site/markdown/ddl-of-carbondata.md index 933a448..965f11c 100644 --- a/src/site/markdown/ddl-of-carbondata.md +++ b/src/site/markdown/ddl-of-carbondata.md @@ -33,7 +33,9 @@ CarbonData DDL statements are documented here,which includes: * [Hive/Parquet folder Structure](#support-flat-folder-same-as-hiveparquet) * [Extra Long String columns](#string-longer-than-32000-characters) * [Compression for Table](#compression-for-table) - * [Bad Records Path](#bad-records-path) + * [Bad Records Path](#bad-records-path) + * [Load Minimum Input File Size](#load-minimum-data-size) + * [CREATE TABLE AS SELECT](#create-table-as-select) * [CREATE EXTERNAL TABLE](#create-external-table) * [External Table on Transactional table location](#create-external-table-on-managed-table-data-location) @@ -84,6 +86,7 @@ CarbonData DDL statements are documented here,which includes: | ------------------------------------------------------------ | ------------------------------------------------------------ | | [DICTIONARY_INCLUDE](#dictionary-encoding-configuration) | Columns for which dictionary needs to be generated | | [NO_INVERTED_INDEX](#inverted-index-configuration) | Columns to exclude from inverted index generation | +| [INVERTED_INDEX](#inverted-index-configuration) | Columns to include for inverted index generation | | [SORT_COLUMNS](#sort-columns-configuration) | Columns to include in sort and its order of sort | | [SORT_SCOPE](#sort-scope-configuration) | Sort scope of the load.Options include no sort, local sort ,batch sort and global sort | | [TABLE_BLOCKSIZE](#table-block-size-configuration) | Size of blocks to write onto hdfs | @@ -104,6 +107,7 @@ CarbonData DDL statements are documented here,which includes: | [LONG_STRING_COLUMNS](#string-longer-than-32000-characters) | Columns which are greater than 32K characters | | [BUCKETNUMBER](#bucketing) | Number of buckets to be created | | [BUCKETCOLUMNS](#bucketing) | Columns which are to be placed in buckets | +| [LOAD_MIN_SIZE_INMB](#load-minimum-data-size) | Minimum input data size per node for data loading | Following are the guidelines for TBLPROPERTIES, CarbonData's additional table options can be set via carbon.properties. @@ -120,11 +124,11 @@ CarbonData DDL statements are documented here,which includes: - ##### Inverted Index Configuration - By default inverted index is enabled, it might help to improve compression ratio and query speed, especially for low cardinality columns which are in reward position. + By default inverted index is disabled as store size will be reduced, it can be enabled by using a table property. It might help to improve compression ratio and query speed, especially for low cardinality columns which are in reward position. Suggested use cases : For high cardinality columns, you can disable the inverted index for improving the data loading performance. ``` - TBLPROPERTIES ('NO_INVERTED_INDEX'='column1, column3') + TBLPROPERTIES ('NO_INVERTED_INDEX'='column1', 'INVERTED_INDEX'='column2, column3') ``` - ##### Sort Columns Configuration @@ -245,7 +249,8 @@ CarbonData DDL statements are documented here,which includes: * TIMESTAMP * DATE * BOOLEAN - + * FLOAT + * BYTE * In case of multi-level complex dataType columns, primitive string/varchar/char columns are considered for local dictionary generation. System Level Properties for Local Dictionary: @@ -445,7 +450,7 @@ CarbonData DDL statements are documented here,which includes: - ##### Compression for table Data compression is also supported by CarbonData. - By default, Snappy is used to compress the data. CarbonData also support ZSTD compressor. + By default, Snappy is used to compress the data. CarbonData also supports ZSTD compressor. User can specify the compressor in the table property: ``` @@ -474,7 +479,19 @@ CarbonData DDL statements are documented here,which includes: be later viewed in table description for reference. ``` - TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords'') + TBLPROPERTIES('BAD_RECORD_PATH'='/opt/badrecords') + ``` + + - ##### Load minimum data size + This property indicates the minimum input data size per node for data loading. + By default it is not enabled. Setting a non-zero integer value will enable this feature. + This property is useful if you have a large cluster and only want a small portion of the nodes to process data loading. + For example, if you have a cluster with 10 nodes and the input data is about 1GB. Without this property, each node will process about 100MB input data and result in at least 10 data files. With this property configured with 512, only 2 nodes will be chosen to process the input data, each with about 512MB input and result in about 2 or 4 files based on the compress ratio. + Moreover, this property can also be specified in the load option. + Notice that once you enable this feature, for load balance, carbondata will ignore the data locality while assigning input data to nodes, this will cause more network traffic. + + ``` + TBLPROPERTIES('LOAD_MIN_SIZE_INMB'='256') ``` ## CREATE TABLE AS SELECT @@ -540,7 +557,7 @@ CarbonData DDL statements are documented here,which includes: ### Create external table on Non-Transactional table data location. Non-Transactional table data location will have only carbondata and carbonindex files, there will not be a metadata folder (table status and schema). - Our SDK module currently support writing data in this format. + Our SDK module currently supports writing data in this format. **Example:** ``` @@ -550,7 +567,7 @@ CarbonData DDL statements are documented here,which includes: ``` Here writer path will have carbondata and index files. - This can be SDK output. Refer [SDK Guide](./sdk-guide.md). + This can be SDK output or C++ SDK output. Refer [SDK Guide](./sdk-guide.md) and [C++ SDK Guide](./csdk-guide.md). **Note:** 1. Dropping of the external table should not delete the files present in the location. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/dml-of-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/dml-of-carbondata.md b/src/site/markdown/dml-of-carbondata.md index 393ebd3..65654a4 100644 --- a/src/site/markdown/dml-of-carbondata.md +++ b/src/site/markdown/dml-of-carbondata.md @@ -58,7 +58,7 @@ CarbonData DML statements are documented here,which includes: | [COLUMNDICT](#columndict) | Path to read the dictionary data from for particular column | | [DATEFORMAT](#dateformattimestampformat) | Format of date in the input csv file | | [TIMESTAMPFORMAT](#dateformattimestampformat) | Format of timestamp in the input csv file | -| [SORT_COLUMN_BOUNDS](#sort-column-bounds) | How to parititon the sort columns to make the evenly distributed | +| [SORT_COLUMN_BOUNDS](#sort-column-bounds) | How to partition the sort columns to make the evenly distributed | | [SINGLE_PASS](#single_pass) | When to enable single pass data loading | | [BAD_RECORDS_LOGGER_ENABLE](#bad-records-handling) | Whether to enable bad records logging | | [BAD_RECORD_PATH](#bad-records-handling) | Bad records logging path. Useful when bad record logging is enabled | @@ -83,7 +83,7 @@ CarbonData DML statements are documented here,which includes: ``` - ##### COMMENTCHAR: - Comment Characters can be provided in the load command if user want to comment lines. + Comment Characters can be provided in the load command if user wants to comment lines. ``` OPTIONS('COMMENTCHAR'='#') ``` @@ -184,7 +184,7 @@ CarbonData DML statements are documented here,which includes: **NOTE:** * SORT_COLUMN_BOUNDS will be used only when the SORT_SCOPE is 'local_sort'. - * Carbondata will use these bounds as ranges to process data concurrently during the final sort percedure. The records will be sorted and written out inside each partition. Since the partition is sorted, all records will be sorted. + * Carbondata will use these bounds as ranges to process data concurrently during the final sort procedure. The records will be sorted and written out inside each partition. Since the partition is sorted, all records will be sorted. * Since the actual order and literal order of the dictionary column are not necessarily the same, we do not recommend you to use this feature if the first sort column is 'dictionary_include'. * The option works better if your CPU usage during loading is low. If your current system CPU usage is high, better not to use this option. Besides, it depends on the user to specify the bounds. If user does not know the exactly bounds to make the data distributed evenly among the bounds, loading performance will still be better than before or at least the same as before. * Users can find more information about this option in the description of PR1953. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/documentation.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/documentation.md b/src/site/markdown/documentation.md index 1b6726a..4a1d176 100644 --- a/src/site/markdown/documentation.md +++ b/src/site/markdown/documentation.md @@ -31,7 +31,7 @@ Apache CarbonData is a new big data file format for faster interactive query usi **CarbonData SQL Language Reference:** CarbonData extends the Spark SQL language and adds several [DDL](./ddl-of-carbondata.md) and [DML](./dml-of-carbondata.md) statements to support operations on it.Refer to the [Reference Manual](./language-manual.md) to understand the supported features and functions. -**Programming Guides:** You can read our guides about [APIs supported](./sdk-guide.md) to learn how to integrate CarbonData with your applications. +**Programming Guides:** You can read our guides about [Java APIs supported](./sdk-guide.md) or [C++ APIs supported](./csdk-guide.md) to learn how to integrate CarbonData with your applications. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/faq.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/faq.md b/src/site/markdown/faq.md index 3ac9a0a..dbcda4f 100644 --- a/src/site/markdown/faq.md +++ b/src/site/markdown/faq.md @@ -216,20 +216,18 @@ TimeZone.setDefault(TimeZone.getTimeZone("Asia/Shanghai")) ## How to check LRU cache memory footprint? To observe the LRU cache memory footprint in the logs, configure the below properties in log4j.properties file. ``` -log4j.logger.org.apache.carbondata.core.memory.UnsafeMemoryManager = DEBUG log4j.logger.org.apache.carbondata.core.cache.CarbonLRUCache = DEBUG ``` -These properties will enable the DEBUG log for the CarbonLRUCache and UnsafeMemoryManager which will print the information of memory consumed using which the LRU cache size can be decided. **Note:** Enabling the DEBUG log will degrade the query performance. +This property will enable the DEBUG log for the CarbonLRUCache and UnsafeMemoryManager which will print the information of memory consumed using which the LRU cache size can be decided. **Note:** Enabling the DEBUG log will degrade the query performance. Ensure carbon.max.driver.lru.cache.size is configured to observe the current cache size. **Example:** ``` -18/09/26 15:05:28 DEBUG UnsafeMemoryManager: pool-44-thread-1 Memory block (org.apache.carbondata.core.memory.MemoryBlock@21312095) is created with size 10. Total memory used 413Bytes, left 536870499Bytes 18/09/26 15:05:29 DEBUG CarbonLRUCache: main Required size for entry /home/target/store/default/stored_as_carbondata_table/Fact/Part0/Segment_0/0_1537954529044.carbonindexmerge :: 181 Current cache size :: 0 -18/09/26 15:05:30 DEBUG UnsafeMemoryManager: main Freeing memory of size: 105available memory: 536870836 -18/09/26 15:05:30 DEBUG UnsafeMemoryManager: main Freeing memory of size: 76available memory: 536870912 18/09/26 15:05:30 INFO CarbonLRUCache: main Removed entry from InMemory lru cache :: /home/target/store/default/stored_as_carbondata_table/Fact/Part0/Segment_0/0_1537954529044.carbonindexmerge ``` +**Note:** If `Removed entry from InMemory LRU cache` are frequently observed in logs, you may have to increase the configured LRU size. +To observe the LRU cache from heap dump, check the heap used by CarbonLRUCache class. ## Getting tablestatus.lock issues When loading data **Symptom** http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/file-structure-of-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/file-structure-of-carbondata.md b/src/site/markdown/file-structure-of-carbondata.md index 8eacd38..7127f37 100644 --- a/src/site/markdown/file-structure-of-carbondata.md +++ b/src/site/markdown/file-structure-of-carbondata.md @@ -122,8 +122,7 @@ Compared with V2: The blocklet data volume of V2 format defaults to 120,000 line #### Footer format -Footer records each carbondata -All blocklet data distribution information and statistical related metadata information (minmax, startkey/endkey) inside the file. +Footer records each carbondata, all blocklet data distribution information and statistical related metadata information (minmax, startkey/endkey) inside the file.  http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/performance-tuning.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/performance-tuning.md b/src/site/markdown/performance-tuning.md index 6c87ce9..7059605 100644 --- a/src/site/markdown/performance-tuning.md +++ b/src/site/markdown/performance-tuning.md @@ -142,7 +142,7 @@ |-----------|-------------|--------| |carbon.number.of.cores.while.loading|Default: 2. This value should be >= 2|Specifies the number of cores used for data processing during data loading in CarbonData. | |carbon.sort.size|Default: 100000. The value should be >= 100.|Threshold to write local file in sort step when loading data| -|carbon.sort.file.write.buffer.size|Default: 50000.|DataOutputStream buffer. | +|carbon.sort.file.write.buffer.size|Default: 16384.|CarbonData sorts and writes data to intermediate files to limit the memory usage. This configuration determines the buffer size to be used for reading and writing such files. | |carbon.merge.sort.reader.thread|Default: 3 |Specifies the number of cores used for temp file merging during data loading in CarbonData.| |carbon.merge.sort.prefetch|Default: true | You may want set this value to false if you have not enough memory| @@ -168,10 +168,9 @@ | carbon.compaction.level.threshold | spark/carbonlib/carbon.properties | Data loading and Querying | For minor compaction, specifies the number of segments to be merged in stage 1 and number of compacted segments to be merged in stage 2. | Each CarbonData load will create one segment, if every load is small in size it will generate many small files over a period of time impacting the query performance. Configuring this parameter will merge the small segment to one big segment which will sort the data and improve the performance. For Example in one telecommunication scenario, the performance improves about 2 times after minor compaction. | | spark.sql.shuffle.partitions | spark/conf/spark-defaults.conf | Querying | The number of task started when spark shuffle. | The value can be 1 to 2 times as much as the executor cores. In an aggregation scenario, reducing the number from 200 to 32 reduced the query time from 17 to 9 seconds. | | spark.executor.instances/spark.executor.cores/spark.executor.memory | spark/conf/spark-defaults.conf | Querying | The number of executors, CPU cores, and memory used for CarbonData query. | In the bank scenario, we provide the 4 CPUs cores and 15 GB for each executor which can get good performance. This 2 value does not mean more the better. It needs to be configured properly in case of limited resources. For example, In the bank scenario, it has enough CPU 32 cores each node but less memory 64 GB each node. So we cannot give more CPU but less memory. For example, when 4 cores and 12GB for each executor. It sometimes happens GC during the query which impact the query performance very much from the 3 second to more than 15 seconds. In this scenario need to increase the memory or decrease the CPU cores. | -| carbon.detail.batch.size | spark/carbonlib/carbon.properties | Data loading | The buffer size to store records, returned from the block scan. | In limit scenario this parameter is very important. For example your query limit is 1000. But if we set this value to 3000 that means we get 3000 records from scan but spark will only take 1000 rows. So the 2000 remaining are useless. In one Finance test case after we set it to 100, in the limit 1000 scenario the performance increase about 2 times in comparison to if we set this value to 12000. | +| carbon.detail.batch.size | spark/carbonlib/carbon.properties | Querying | The buffer size to store records, returned from the block scan. | In limit scenario this parameter is very important. For example your query limit is 1000. But if we set this value to 3000 that means we get 3000 records from scan but spark will only take 1000 rows. So the 2000 remaining are useless. In one Finance test case after we set it to 100, in the limit 1000 scenario the performance increase about 2 times in comparison to if we set this value to 12000. | | carbon.use.local.dir | spark/carbonlib/carbon.properties | Data loading | Whether use YARN local directories for multi-table load disk load balance | If this is set it to true CarbonData will use YARN local directories for multi-table load disk load balance, that will improve the data load performance. | -| carbon.use.multiple.temp.dir | spark/carbonlib/carbon.properties | Data loading | Whether to use multiple YARN local directories during table data loading for disk load balance | After enabling 'carbon.use.local.dir', if this is set to true, CarbonData will use all YARN local directories during data load for disk load balance, that will improve the data load performance. Please enable this property when you encounter disk hotspot problem during data loading. | -| carbon.sort.temp.compressor | spark/carbonlib/carbon.properties | Data loading | Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading. | The optional values are 'SNAPPY','GZIP','BZIP2','LZ4','ZSTD', and empty. By default, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck. | +| carbon.sort.temp.compressor | spark/carbonlib/carbon.properties | Data loading | Specify the name of compressor to compress the intermediate sort temporary files during sort procedure in data loading. | The optional values are 'SNAPPY','GZIP','BZIP2','LZ4','ZSTD', and empty. Specially, empty means that Carbondata will not compress the sort temp files. This parameter will be useful if you encounter disk bottleneck. | | carbon.load.skewedDataOptimization.enabled | spark/carbonlib/carbon.properties | Data loading | Whether to enable size based block allocation strategy for data loading. | When loading, carbondata will use file size based block allocation strategy for task distribution. It will make sure that all the executors process the same size of data -- It's useful if the size of your input data files varies widely, say 1MB to 1GB. | | carbon.load.min.size.enabled | spark/carbonlib/carbon.properties | Data loading | Whether to enable node minumun input data size allocation strategy for data loading.| When loading, carbondata will use node minumun input data size allocation strategy for task distribution. It will make sure the nodes load the minimum amount of data -- It's useful if the size of your input data files very small, say 1MB to 256MB,Avoid generating a large number of small files. | http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/quick-start-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/quick-start-guide.md b/src/site/markdown/quick-start-guide.md index fd535ae..a14b1cd 100644 --- a/src/site/markdown/quick-start-guide.md +++ b/src/site/markdown/quick-start-guide.md @@ -294,7 +294,7 @@ hdfs://<host_name>:port/user/hive/warehouse/carbon.store ## Installing and Configuring CarbonData on Presto **NOTE:** **CarbonData tables cannot be created nor loaded from Presto. User need to create CarbonData Table and load data into it -either with [Spark](#installing-and-configuring-carbondata-to-run-locally-with-spark-shell) or [SDK](./sdk-guide.md). +either with [Spark](#installing-and-configuring-carbondata-to-run-locally-with-spark-shell) or [SDK](./sdk-guide.md) or [C++ SDK](./csdk-guide.md). Once the table is created,it can be queried from Presto.** http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/sdk-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/sdk-guide.md b/src/site/markdown/sdk-guide.md index be42b3f..8abc3b1 100644 --- a/src/site/markdown/sdk-guide.md +++ b/src/site/markdown/sdk-guide.md @@ -24,11 +24,15 @@ CarbonData provides SDK to facilitate # SDK Writer -In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader. +In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader. +If user want to use SDK, except carbondata-store-sdk-x.x.x-SNAPSHOT.jar, +it needs carbondata-core-x.x.x-SNAPSHOT.jar, carbondata-common-x.x.x-SNAPSHOT.jar, +carbondata-format-x.x.x-SNAPSHOT.jar, carbondata-hadoop-x.x.x-SNAPSHOT.jar and carbondata-processing-x.x.x-SNAPSHOT.jar. +What's more, user also can use carbondata-sdk.jar directly. This SDK writer, writes carbondata file and carbonindex file at a given path. External client can make use of this writer to convert other format data or live data to create carbondata and index files. -These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present. +These SDK writer output contains just carbondata and carbonindex files. No metadata folder will be present. ## Quick example @@ -67,7 +71,7 @@ These SDK writer output contains just a carbondata and carbonindex files. No met CarbonProperties.getInstance().addProperty("enable.offheap.sort", enableOffheap); - CarbonWriterBuilder builder = CarbonWriter.builder().outputPath(path).withCsvInput(schema); + CarbonWriterBuilder builder = CarbonWriter.builder().outputPath(path).withCsvInput(schema).writtenBy("SDK"); CarbonWriter writer = builder.build(); @@ -124,7 +128,7 @@ public class TestSdkAvro { try { CarbonWriter writer = CarbonWriter.builder() .outputPath(path) - .withAvroInput(new org.apache.avro.Schema.Parser().parse(avroSchema)).build(); + .withAvroInput(new org.apache.avro.Schema.Parser().parse(avroSchema)).writtenBy("SDK").build(); for (int i = 0; i < 100; i++) { writer.write(record); @@ -164,7 +168,7 @@ public class TestSdkJson { Schema CarbonSchema = new Schema(fields); - CarbonWriterBuilder builder = CarbonWriter.builder().outputPath(path).withJsonInput(CarbonSchema); + CarbonWriterBuilder builder = CarbonWriter.builder().outputPath(path).withJsonInput(CarbonSchema).writtenBy("SDK"); // initialize json writer with carbon schema CarbonWriter writer = builder.build(); @@ -371,6 +375,8 @@ public CarbonWriterBuilder withLoadOptions(Map<String, String> options); * j. sort_scope -- "local_sort", "no_sort", "batch_sort". default value is "local_sort" * k. long_string_columns -- comma separated string columns which are more than 32k length. * default value is null. +* l. inverted_index -- comma separated string columns for which inverted index needs to be +* generated * * @return updated CarbonWriterBuilder */ @@ -400,6 +406,17 @@ public CarbonWriterBuilder withHadoopConf(Configuration conf) ``` ``` + /** + * Updates the hadoop configuration with the given key value + * + * @param key key word + * @param value value + * @return this object + */ + public CarbonWriterBuilder withHadoopConf(String key, String value); +``` + +``` /** * to build a {@link CarbonWriter}, which accepts row in CSV format * @@ -431,6 +448,27 @@ public CarbonWriterBuilder withJsonInput(Schema carbonSchema); ``` /** +* To support writing the ApplicationName which is writing the carbondata file +* This is a mandatory API to call, else the build() call will fail with error. +* @param application name which is writing the carbondata files +* @return CarbonWriterBuilder +*/ +public CarbonWriterBuilder writtenBy(String appName) { +``` + +``` +/** +* sets the list of columns for which inverted index needs to generated +* @param invertedIndexColumns is a string array of columns for which inverted index needs to +* generated. +* If it is null or an empty array, inverted index will be generated for none of the columns +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder invertedIndexFor(String[] invertedIndexColumns); +``` + +``` +/** * Build a {@link CarbonWriter} * This writer is not thread safe, * use withThreadSafe() configuration in multi thread environment @@ -442,7 +480,25 @@ public CarbonWriterBuilder withJsonInput(Schema carbonSchema); public CarbonWriter build() throws IOException, InvalidLoadOptionException; ``` +``` + /** + * Configure Row Record Reader for reading. + * + */ + public CarbonReaderBuilder withRowRecordReader() +``` + ### Class org.apache.carbondata.sdk.file.CarbonWriter + +``` +/** +* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} +*/ +public static CarbonWriterBuilder builder() { + return new CarbonWriterBuilder(); +} +``` + ``` /** * Write an object to the file, the format of the object depends on the implementation @@ -463,15 +519,6 @@ public abstract void write(Object object) throws IOException; public abstract void close() throws IOException; ``` -``` -/** -* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} -*/ -public static CarbonWriterBuilder builder() { - return new CarbonWriterBuilder(); -} -``` - ### Class org.apache.carbondata.sdk.file.Field ``` /** @@ -581,6 +628,26 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ ``` ``` +/** + * Breaks the list of CarbonRecordReader in CarbonReader into multiple + * CarbonReader objects, each iterating through some 'carbondata' files + * and return that list of CarbonReader objects + * + * If the no. of files is greater than maxSplits, then break the + * CarbonReader into maxSplits splits, with each split iterating + * through >= 1 file. + * + * If the no. of files is less than maxSplits, then return list of + * CarbonReader with size as the no. of files, with each CarbonReader + * iterating through exactly one file + * + * @param maxSplits: Int + * @return list of CarbonReader objects + */ + public List<CarbonReader> split(int maxSplits); +``` + +``` /** * Return true if has next row */ @@ -596,6 +663,13 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ ``` /** + * Read and return next batch row objects + */ + public Object[] readNextBatchRow(); +``` + +``` + /** * Close reader */ public void close(); @@ -633,6 +707,16 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ ``` ``` + /** + * Sets the batch size of records to read + * + * @param batch batch size + * @return updated CarbonReaderBuilder + */ + public CarbonReaderBuilder withBatch(int batch); +``` + +``` /** * To support hadoop configuration * @@ -643,6 +727,17 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ ``` ``` + /** + * Updates the hadoop configuration with the given key value + * + * @param key key word + * @param value value + * @return this object + */ + public CarbonReaderBuilder withHadoopConf(String key, String value); +``` + +``` /** * Build CarbonReader * @@ -662,6 +757,7 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ * @return schema object * @throws IOException */ + @Deprecated public static Schema readSchemaInSchemaFile(String schemaFilePath); ``` @@ -672,6 +768,7 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ * @param dataFilePath complete path including carbondata file name * @return Schema object */ + @Deprecated public static Schema readSchemaInDataFile(String dataFilePath); ``` @@ -683,9 +780,49 @@ Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/ * @return schema object * @throws IOException */ + @Deprecated public static Schema readSchemaInIndexFile(String indexFilePath); ``` +``` + /** + * read schema from path, + * path can be folder path,carbonindex file path, and carbondata file path + * and will not check all files schema + * + * @param path file/folder path + * @return schema + * @throws IOException + */ + public static Schema readSchema(String path); +``` + +``` + /** + * read schema from path, + * path can be folder path,carbonindex file path, and carbondata file path + * and user can decide whether check all files schema + * + * @param path file/folder path + * @param validateSchema whether check all files schema + * @return schema + * @throws IOException + */ + public static Schema readSchema(String path, boolean validateSchema); +``` + +``` + /** + * This method return the version details in formatted string by reading from carbondata file + * If application name is SDK_1.0.0 and this has written the carbondata file in carbondata 1.6 project version, + * then this API returns the String "SDK_1.0.0 in version: 1.6.0-SNAPSHOT" + * @param dataFilePath complete path including carbondata file name + * @return string with information of who has written this file in which carbondata project version + * @throws IOException + */ + public static String getVersionDetails(String dataFilePath); +``` + ### Class org.apache.carbondata.sdk.file.Schema ``` /** http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/streaming-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/streaming-guide.md b/src/site/markdown/streaming-guide.md index 714b07a..0987ed2 100644 --- a/src/site/markdown/streaming-guide.md +++ b/src/site/markdown/streaming-guide.md @@ -31,9 +31,10 @@ - [StreamSQL](#streamsql) - [Defining Streaming Table](#streaming-table) - [Streaming Job Management](#streaming-job-management) - - [START STREAM](#start-stream) - - [STOP STREAM](#stop-stream) + - [CREATE STREAM](#create-stream) + - [DROP STREAM](#drop-stream) - [SHOW STREAMS](#show-streams) + - [CLOSE STREAM](#close-stream) ## Quick example Download and unzip spark-2.2.0-bin-hadoop2.7.tgz, and export $SPARK_HOME @@ -333,7 +334,7 @@ Following example shows how to start a streaming ingest job sql( """ - |START STREAM job123 ON TABLE sink + |CREATE STREAM job123 ON TABLE sink |STMPROPERTIES( | 'trigger'='ProcessingTime', | 'interval'='1 seconds') @@ -343,7 +344,7 @@ Following example shows how to start a streaming ingest job | WHERE id % 2 = 1 """.stripMargin) - sql("STOP STREAM job123") + sql("DROP STREAM job123") sql("SHOW STREAMS [ON TABLE tableName]") ``` @@ -360,13 +361,13 @@ These two tables are normal carbon tables, they can be queried independently. As above example shown: -- `START STREAM jobName ON TABLE tableName` is used to start a streaming ingest job. -- `STOP STREAM jobName` is used to stop a streaming job by its name +- `CREATE STREAM jobName ON TABLE tableName` is used to start a streaming ingest job. +- `DROP STREAM jobName` is used to stop a streaming job by its name - `SHOW STREAMS [ON TABLE tableName]` is used to print streaming job information -##### START STREAM +##### CREATE STREAM When this is issued, carbon will start a structured streaming job to do the streaming ingestion. Before launching the job, system will validate: @@ -424,11 +425,25 @@ For Kafka data source, create the source table by: ) ``` +- Then CREATE STREAM can be used to start the streaming ingest job from source table to sink table +``` +CREATE STREAM job123 ON TABLE sink +STMPROPERTIES( + 'trigger'='ProcessingTime', + 'interval'='10 seconds' +) +AS + SELECT * + FROM source + WHERE id % 2 = 1 +``` -##### STOP STREAM - -When this is issued, the streaming job will be stopped immediately. It will fail if the jobName specified is not exist. +##### DROP STREAM +When `DROP STREAM` is issued, the streaming job will be stopped immediately. It will fail if the jobName specified is not exist. +``` +DROP STREAM job123 +``` ##### SHOW STREAMS @@ -441,4 +456,9 @@ When this is issued, the streaming job will be stopped immediately. It will fail `SHOW STREAMS` command will show all stream jobs in the system. +##### ALTER TABLE CLOSE STREAM + +When the streaming application is stopped, and user want to manually trigger data conversion from carbon streaming files to columnar files, one can use +`ALTER TABLE sink COMPACT 'CLOSE_STREAMING';` + http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/ae77df2e/src/site/markdown/usecases.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/usecases.md b/src/site/markdown/usecases.md index e8b98b5..c029bb3 100644 --- a/src/site/markdown/usecases.md +++ b/src/site/markdown/usecases.md @@ -72,7 +72,6 @@ Apart from these, the following CarbonData configuration was suggested to be con | Data Loading | table_blocksize | 256 | To efficiently schedule multiple tasks during query | | Data Loading | carbon.sort.intermediate.files.limit | 100 | Increased to 100 as number of cores are more.Can perform merging in backgorund.If less number of files to merge, sort threads would be idle | | Data Loading | carbon.use.local.dir | TRUE | yarn application directory will be usually on a single disk.YARN would be configured with multiple disks to be used as temp or to assign randomly to applications. Using the yarn temp directory will allow carbon to use multiple disks and improve IO performance | -| Data Loading | carbon.use.multiple.temp.dir | TRUE | multiple disks to write sort files will lead to better IO and reduce the IO bottleneck | | Compaction | carbon.compaction.level.threshold | 6,6 | Since frequent small loads, compacting more segments will give better query results | | Compaction | carbon.enable.auto.load.merge | true | Since data loading is small,auto compacting keeps the number of segments less and also compaction can complete in time | | Compaction | carbon.number.of.cores.while.compacting | 4 | Higher number of cores can improve the compaction speed | @@ -127,7 +126,6 @@ Use all columns are no-dictionary as the cardinality is high. | Data Loading | table_blocksize | 512 | To efficiently schedule multiple tasks during query. This size depends on data scenario.If data is such that the filters would select less number of blocklets to scan, keeping higher number works well.If the number blocklets to scan is more, better to reduce the size as more tasks can be scheduled in parallel. | | Data Loading | carbon.sort.intermediate.files.limit | 100 | Increased to 100 as number of cores are more.Can perform merging in backgorund.If less number of files to merge, sort threads would be idle | | Data Loading | carbon.use.local.dir | TRUE | yarn application directory will be usually on a single disk.YARN would be configured with multiple disks to be used as temp or to assign randomly to applications. Using the yarn temp directory will allow carbon to use multiple disks and improve IO performance | -| Data Loading | carbon.use.multiple.temp.dir | TRUE | multiple disks to write sort files will lead to better IO and reduce the IO bottleneck | | Data Loading | sort.inmemory.size.in.mb | 92160 | Memory allocated to do inmemory sorting. When more memory is available in the node, configuring this will retain more sort blocks in memory so that the merge sort is faster due to no/very less IO | | Compaction | carbon.major.compaction.size | 921600 | Sum of several loads to combine into single segment | | Compaction | carbon.number.of.cores.while.compacting | 12 | Higher number of cores can improve the compaction speed.Data size is huge.Compaction need to use more threads to speed up the process |
