Repository: carbondata-site Updated Branches: refs/heads/asf-site 8cb979c8d -> 61436c8e4
http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/markdown/data-management-on-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/data-management-on-carbondata.md b/src/site/markdown/data-management-on-carbondata.md index 51e98ab..706209c 100644 --- a/src/site/markdown/data-management-on-carbondata.md +++ b/src/site/markdown/data-management-on-carbondata.md @@ -35,11 +35,11 @@ This tutorial is going to introduce all commands and data operations on CarbonDa ``` CREATE TABLE [IF NOT EXISTS] [db_name.]table_name[(col_name data_type , ...)] - STORED BY 'carbondata' + STORED AS carbondata [TBLPROPERTIES (property_name=property_value, ...)] [LOCATION 'path'] ``` - **NOTE:** CarbonData also supports "STORED AS carbondata". Find example code at [CarbonSessionExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala) in the CarbonData repo. + **NOTE:** CarbonData also supports "STORED AS carbondata" and "USING carbondata". Find example code at [CarbonSessionExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala) in the CarbonData repo. ### Usage Guidelines Following are the guidelines for TBLPROPERTIES, CarbonData's additional table options can be set via carbon.properties. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/markdown/lucene-datamap-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/lucene-datamap-guide.md b/src/site/markdown/lucene-datamap-guide.md new file mode 100644 index 0000000..5f7a2e4 --- /dev/null +++ b/src/site/markdown/lucene-datamap-guide.md @@ -0,0 +1,159 @@ +# CarbonData Lucene DataMap (Alpha feature in 1.4.0) + +* [DataMap Management](#datamap-management) +* [Lucene Datamap](#lucene-datamap-introduction) +* [Loading Data](#loading-data) +* [Querying Data](#querying-data) +* [Data Management](#data-management-with-lucene-datamap) + +#### DataMap Management +Lucene DataMap can be created using following DDL + ``` + CREATE DATAMAP [IF NOT EXISTS] datamap_name + ON TABLE main_table + USING 'lucene' + DMPROPERTIES ('index_columns'='city, name', ...) + ``` + +DataMap can be dropped using following DDL: + ``` + DROP DATAMAP [IF EXISTS] datamap_name + ON TABLE main_table + ``` +To show all DataMaps created, use: + ``` + SHOW DATAMAP + ON TABLE main_table + ``` +It will show all DataMaps created on main table. + + +## Lucene DataMap Introduction + Lucene is a high performance, full featured text search engine. Lucene is integrated to carbon as + an index datamap and managed along with main tables by CarbonData.User can create lucene datamap + to improve query performance on string columns which has content of more length. So, user can + search tokenized word or pattern of it using lucene query on text content. + + For instance, main table called **datamap_test** which is defined as: + + ``` + CREATE TABLE datamap_test ( + name string, + age int, + city string, + country string) + STORED BY 'carbondata' + ``` + + User can create Lucene datamap using the Create DataMap DDL: + + ``` + CREATE DATAMAP dm + ON TABLE datamap_test + USING 'lucene' + DMPROPERTIES ('INDEX_COLUMNS' = 'name, country',) + ``` + +**DMProperties** +1. INDEX_COLUMNS: The list of string columns on which lucene creates indexes. +2. FLUSH_CACHE: size of the cache to maintain in Lucene writer, if specified then it tries to + aggregate the unique data till the cache limit and flush to Lucene. It is best suitable for low + cardinality dimensions. +3. SPLIT_BLOCKLET: when made as true then store the data in blocklet wise in lucene , it means new + folder will be created for each blocklet, thus, it eliminates storing blockletid in lucene and + also it makes lucene small chunks of data. + +## Loading data +When loading data to main table, lucene index files will be generated for all the +index_columns(String Columns) given in DMProperties which contains information about the data +location of index_columns. These index files will be written inside a folder named with datamap name +inside each segment folders. + +A system level configuration carbon.lucene.compression.mode can be added for best compression of +lucene index files. The default value is speed, where the index writing speed will be more. If the +value is compression, the index file size will be compressed. + +## Querying data +As a technique for query acceleration, Lucene indexes cannot be queried directly. +Queries are to be made on main table. when a query with TEXT_MATCH('name:c10') or +TEXT_MATCH_WITH_LIMIT('name:n10',10)[the second parameter represents the number of result to be +returned, if user does not specify this value, all results will be returned without any limit] is +fired, two jobs are fired.The first job writes the temporary files in folder created at table level +which contains lucene's seach results and these files will be read in second job to give faster +results. These temporary files will be cleared once the query finishes. + +User can verify whether a query can leverage Lucene datamap or not by executing `EXPLAIN` +command, which will show the transformed logical plan, and thus user can check whether TEXT_MATCH() +filter is applied on query or not. + +**Note:** + 1. The filter columns in TEXT_MATCH or TEXT_MATCH_WITH_LIMIT must be always in lower case and +filter condition like 'AND','OR' must be in upper case. + + Ex: + ``` + select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*') + ``` + +2. Query supports only one TEXT_MATCH udf for filter condition and not multiple udfs. + + The following query is supported: + ``` + select * from datamap_test where TEXT_MATCH('name:*10 AND name:*n*') + ``` + + The following query is not supported: + ``` + select * from datamap_test where TEXT_MATCH('name:*10) AND TEXT_MATCH(name:*n*') + ``` + + +Below like queries can be converted to text_match queries as following: +``` +select * from datamap_test where name='n10' + +select * from datamap_test where name like 'n1%' + +select * from datamap_test where name like '%10' + +select * from datamap_test where name like '%n%' + +select * from datamap_test where name like '%10' and name not like '%n%' +``` +Lucene TEXT_MATCH Queries: +``` +select * from datamap_test where TEXT_MATCH('name:n10') + +select * from datamap_test where TEXT_MATCH('name:n1*') + +select * from datamap_test where TEXT_MATCH('name:*10') + +select * from datamap_test where TEXT_MATCH('name:*n*') + +select * from datamap_test where TEXT_MATCH('name:*10 -name:*n*') +``` +**Note:** For lucene queries and syntax, refer to [lucene-syntax](www.lucenetutorial.com/lucene-query-syntax.html) + +## Data Management with lucene datamap +Once there is lucene datamap is created on the main table, following command on the main +table +is not supported: +1. Data management command: `UPDATE/DELETE`. +2. Schema management command: `ALTER TABLE DROP COLUMN`, `ALTER TABLE CHANGE DATATYPE`, +`ALTER TABLE RENAME`. + +**Note**: Adding a new column is supported, and for dropping columns and change datatype +command, CarbonData will check whether it will impact the lucene datamap, if not, the operation +is allowed, otherwise operation will be rejected by throwing exception. + + +3. Partition management command: `ALTER TABLE ADD/DROP PARTITION`. + +However, there is still way to support these operations on main table, in current CarbonData +release, user can do as following: +1. Remove the lucene datamap by `DROP DATAMAP` command. +2. Carry out the data management operation on main table. +3. Create the lucene datamap again by `CREATE DATAMAP` command. +Basically, user can manually trigger the operation by re-building the datamap. + + http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/markdown/sdk-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/sdk-guide.md b/src/site/markdown/sdk-guide.md new file mode 100644 index 0000000..1d225a9 --- /dev/null +++ b/src/site/markdown/sdk-guide.md @@ -0,0 +1,680 @@ +# SDK Guide +In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar, including SDK writer and reader. +# SDK Writer +This SDK writer, writes carbondata file and carbonindex file at a given path. +External client can make use of this writer to convert other format data or live data to create carbondata and index files. +These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present. + +## Quick example + +### Example with csv format + +```java + import java.io.IOException; + + import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException; + import org.apache.carbondata.core.metadata.datatype.DataTypes; + import org.apache.carbondata.core.util.CarbonProperties; + import org.apache.carbondata.sdk.file.CarbonWriter; + import org.apache.carbondata.sdk.file.CarbonWriterBuilder; + import org.apache.carbondata.sdk.file.Field; + import org.apache.carbondata.sdk.file.Schema; + + public class TestSdk { + + // pass true or false while executing the main to use offheap memory or not + public static void main(String[] args) throws IOException, InvalidLoadOptionException { + if (args.length > 0 && args[0] != null) { + testSdkWriter(args[0]); + } else { + testSdkWriter("true"); + } + } + + public static void testSdkWriter(String enableOffheap) throws IOException, InvalidLoadOptionException { + String path = "./target/testCSVSdkWriter"; + + Field[] fields = new Field[2]; + fields[0] = new Field("name", DataTypes.STRING); + fields[1] = new Field("age", DataTypes.INT); + + Schema schema = new Schema(fields); + + CarbonProperties.getInstance().addProperty("enable.offheap.sort", enableOffheap); + + CarbonWriterBuilder builder = CarbonWriter.builder().outputPath(path); + + CarbonWriter writer = builder.buildWriterForCSVInput(schema); + + int rows = 5; + for (int i = 0; i < rows; i++) { + writer.write(new String[] { "robot" + (i % 10), String.valueOf(i) }); + } + writer.close(); + } + } +``` + +### Example with Avro format +```java +import java.io.IOException; + +import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException; +import org.apache.carbondata.core.metadata.datatype.DataTypes; +import org.apache.carbondata.sdk.file.AvroCarbonWriter; +import org.apache.carbondata.sdk.file.CarbonWriter; +import org.apache.carbondata.sdk.file.Field; + +import org.apache.avro.generic.GenericData; +import org.apache.commons.lang.CharEncoding; + +import tech.allegro.schema.json2avro.converter.JsonAvroConverter; + +public class TestSdkAvro { + + public static void main(String[] args) throws IOException, InvalidLoadOptionException { + testSdkWriter(); + } + + + public static void testSdkWriter() throws IOException, InvalidLoadOptionException { + String path = "./AvroCarbonWriterSuiteWriteFiles"; + // Avro schema + String avroSchema = + "{" + + " \"type\" : \"record\"," + + " \"name\" : \"Acme\"," + + " \"fields\" : [" + + "{ \"name\" : \"fname\", \"type\" : \"string\" }," + + "{ \"name\" : \"age\", \"type\" : \"int\" }]" + + "}"; + + String json = "{\"fname\":\"bob\", \"age\":10}"; + + // conversion to GenericData.Record + JsonAvroConverter converter = new JsonAvroConverter(); + GenericData.Record record = converter.convertToGenericDataRecord( + json.getBytes(CharEncoding.UTF_8), new org.apache.avro.Schema.Parser().parse(avroSchema)); + + try { + CarbonWriter writer = CarbonWriter.builder() + .outputPath(path) + .buildWriterForAvroInput(new org.apache.avro.Schema.Parser().parse(avroSchema)); + + for (int i = 0; i < 100; i++) { + writer.write(record); + } + writer.close(); + } catch (Exception e) { + e.printStackTrace(); + } + } +} +``` + +## Datatypes Mapping +Each of SQL data types are mapped into data types of SDK. Following are the mapping: + +| SQL DataTypes | Mapped SDK DataTypes | +|---------------|----------------------| +| BOOLEAN | DataTypes.BOOLEAN | +| SMALLINT | DataTypes.SHORT | +| INTEGER | DataTypes.INT | +| BIGINT | DataTypes.LONG | +| DOUBLE | DataTypes.DOUBLE | +| VARCHAR | DataTypes.STRING | +| DATE | DataTypes.DATE | +| TIMESTAMP | DataTypes.TIMESTAMP | +| STRING | DataTypes.STRING | +| DECIMAL | DataTypes.createDecimalType(precision, scale) | + + +## API List + +### Class org.apache.carbondata.sdk.file.CarbonWriterBuilder +``` +/** +* Sets the output path of the writer builder +* @param path is the absolute path where output files are written +* This method must be called when building CarbonWriterBuilder +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder outputPath(String path); +``` + +``` +/** +* If set false, writes the carbondata and carbonindex files in a flat folder structure +* @param isTransactionalTable is a boolelan value +* if set to false, then writes the carbondata and carbonindex files +* in a flat folder structure. +* if set to true, then writes the carbondata and carbonindex files +* in segment folder structure.. +* By default set to false. +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder isTransactionalTable(boolean isTransactionalTable); +``` + +``` +/** +* to set the timestamp in the carbondata and carbonindex index files +* @param UUID is a timestamp to be used in the carbondata and carbonindex index files. +* By default set to zero. +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder uniqueIdentifier(long UUID); +``` + +``` +/** +* To set the carbondata file size in MB between 1MB-2048MB +* @param blockSize is size in MB between 1MB to 2048 MB +* default value is 1024 MB +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withBlockSize(int blockSize); +``` + +``` +/** +* To set the blocklet size of carbondata file +* @param blockletSize is blocklet size in MB +* default value is 64 MB +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withBlockletSize(int blockletSize); +``` + +``` +/** +* sets the list of columns that needs to be in sorted order +* @param sortColumns is a string array of columns that needs to be sorted. +* If it is null or by default all dimensions are selected for sorting +* If it is empty array, no columns are sorted +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder sortBy(String[] sortColumns); +``` + +``` +/** +* If set, create a schema file in metadata folder. +* @param persist is a boolean value, If set to true, creates a schema file in metadata folder. +* By default set to false. will not create metadata folder +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder persistSchemaFile(boolean persist); +``` + +``` +/** +* sets the taskNo for the writer. SDKs concurrently running +* will set taskNo in order to avoid conflicts in file's name during write. +* @param taskNo is the TaskNo user wants to specify. +* by default it is system time in nano seconds. +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder taskNo(String taskNo); +``` + +``` +/** +* To support the load options for sdk writer +* @param options key,value pair of load options. +* supported keys values are +* a. bad_records_logger_enable -- true (write into separate logs), false +* b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT +* c. bad_record_path -- path +* d. dateformat -- same as JAVA SimpleDateFormat +* e. timestampformat -- same as JAVA SimpleDateFormat +* f. complex_delimiter_level_1 -- value to Split the complexTypeData +* g. complex_delimiter_level_2 -- value to Split the nested complexTypeData +* h. quotechar +* i. escapechar +* +* Default values are as follows. +* +* a. bad_records_logger_enable -- "false" +* b. bad_records_action -- "FAIL" +* c. bad_record_path -- "" +* d. dateformat -- "" , uses from carbon.properties file +* e. timestampformat -- "", uses from carbon.properties file +* f. complex_delimiter_level_1 -- "$" +* g. complex_delimiter_level_2 -- ":" +* h. quotechar -- "\"" +* i. escapechar -- "\\" +* +* @return updated CarbonWriterBuilder +*/ +public CarbonWriterBuilder withLoadOptions(Map<String, String> options); +``` + +``` +/** +* Build a {@link CarbonWriter}, which accepts row in CSV format object +* @param schema carbon Schema object {org.apache.carbondata.sdk.file.Schema} +* @return CSVCarbonWriter +* @throws IOException +* @throws InvalidLoadOptionException +*/ +public CarbonWriter buildWriterForCSVInput() throws IOException, InvalidLoadOptionException; +``` + +``` +/** +* Build a {@link CarbonWriter}, which accepts Avro format object +* @param avroSchema avro Schema object {org.apache.avro.Schema} +* @return AvroCarbonWriter +* @throws IOException +* @throws InvalidLoadOptionException +*/ +public CarbonWriter buildWriterForAvroInput() throws IOException, InvalidLoadOptionException; +``` + +### Class org.apache.carbondata.sdk.file.CarbonWriter +``` +/** +* Write an object to the file, the format of the object depends on the implementation +* If AvroCarbonWriter, object is of type org.apache.avro.generic.GenericData.Record +* If CSVCarbonWriter, object is of type String[] +* Note: This API is not thread safe +* @param object +* @throws IOException +*/ +public abstract void write(Object object) throws IOException; +``` + +``` +/** +* Flush and close the writer +*/ +public abstract void close() throws IOException; +``` + +``` +/** +* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} +*/ +public static CarbonWriterBuilder builder() { + return new CarbonWriterBuilder(); +} +``` + +### Class org.apache.carbondata.sdk.file.Field +``` +/** +* Field Constructor +* @param name name of the field +* @param type datatype of field, specified in strings. +*/ +public Field(String name, String type); +``` + +``` +/** +* Field constructor +* @param name name of the field +* @param type datatype of the field of class DataType +*/ +public Field(String name, DataType type); +``` + +### Class org.apache.carbondata.sdk.file.Schema + +``` +/** +* construct a schema with fields +* @param fields +*/ +public Schema(Field[] fields); +``` + +``` +/** +* Create a Schema using JSON string, for example: +* [ +* {"name":"string"}, +* {"age":"int"} +* ] +* @param json specified as string +* @return Schema +*/ +public static Schema parseJson(String json); +``` + +### Class org.apache.carbondata.core.util.CarbonProperties + +``` +/** +* This method will be responsible to get the instance of CarbonProperties class +* +* @return carbon properties instance +*/ +public static CarbonProperties getInstance(); +``` + +``` +/** +* This method will be used to add a new property +* +* @param key is a property name to set for carbon. +* @param value is valid parameter corresponding to property. +* @return CarbonProperties object +*/ +public CarbonProperties addProperty(String key, String value); +``` + +``` +/** +* This method will be used to get the property value. If property is not +* present, then it will return the default value. +* +* @param key is a property name to get user specified value. +* @return properties value for corresponding key. If not set, then returns null. +*/ +public String getProperty(String key); +``` + +``` +/** +* This method will be used to get the property value. If property is not +* present, then it will return the default value. +* +* @param key is a property name to get user specified value.. +* @param defaultValue used to be returned by function if corrosponding key not set. +* @return properties value for corresponding key. If not set, then returns specified defaultValue. +*/ +public String getProperty(String key, String defaultValue); +``` +Reference : [list of carbon properties](http://carbondata.apache.org/configuration-parameters.html) + +### Class org.apache.carbondata.sdk.file.AvroCarbonWriter +``` +/** +* converts avro schema to carbon schema, required by carbonWriter +* +* @param avroSchemaString json formatted avro schema as string +* @return carbon sdk schema +*/ +public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); +``` +# SDK Reader +This SDK reader reads CarbonData file and carbonindex file at a given path. +External client can make use of this reader to read CarbonData files without CarbonSession. +## Quick example +``` + // 1. Create carbon reader + String path = "./testWriteFiles"; + CarbonReader reader = CarbonReader + .builder(path, "_temp") + .projection(new String[]{"name", "age"}) + .build(); + + // 2. Read data + int i = 0; + while (reader.hasNext()) { + Object[] row = (Object[]) reader.readNextRow(); + System.out.println(row[0] + "\t" + row[1]); + i++; + } + + // 3. Close this reader + reader.close(); +``` + +Find example code at [CarbonReaderExample](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/CarbonReaderExample.java) in the CarbonData repo. + +## API List + +### Class org.apache.carbondata.sdk.file.CarbonReader +``` + /** + * Return a new CarbonReaderBuilder instance + */ + public static CarbonReaderBuilder builder(String tablePath, String tableName); +``` + +``` + /** + * Return true if has next row + */ + public boolean hasNext(); +``` + +``` + /** + * Read and return next row object + */ + public T readNextRow(); +``` + +``` + /** + * Close reader + */ + public void close(); +``` + +### Class org.apache.carbondata.sdk.file.CarbonReaderBuilder +``` + /** + * Construct a CarbonReaderBuilder with table path and table name + * + * @param tablePath table path + * @param tableName table name + */ + CarbonReaderBuilder(String tablePath, String tableName); +``` + +``` + /** + * Configure the projection column names of carbon reader + * + * @param projectionColumnNames projection column names + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder projection(String[] projectionColumnNames); +``` + +``` + /** + * Project all Columns for carbon reader + * + * @return CarbonReaderBuilder object + * @throws IOException + */ + public CarbonReaderBuilder projectAllColumns(); +``` + +``` + /** + * Configure the transactional status of table + * If set to false, then reads the carbondata and carbonindex files from a flat folder structure. + * If set to true, then reads the carbondata and carbonindex files from segment folder structure. + * Default value is false + * + * @param isTransactionalTable whether is transactional table or not + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder isTransactionalTable(boolean isTransactionalTable); +``` + +``` + /** + * Configure the filter expression for carbon reader + * + * @param filterExpression filter expression + * @return CarbonReaderBuilder object + */ + public CarbonReaderBuilder filter(Expression filterExpression); +``` + +``` + /** + * Set the access key for S3 + * + * @param key the string of access key for different S3 type,like: fs.s3a.access.key + * @param value the value of access key + * @return CarbonWriterBuilder + */ + public CarbonReaderBuilder setAccessKey(String key, String value); +``` + +``` + /** + * Set the access key for S3. + * + * @param value the value of access key + * @return CarbonWriterBuilder object + */ + public CarbonReaderBuilder setAccessKey(String value); +``` + +``` + /** + * Set the secret key for S3 + * + * @param key the string of secret key for different S3 type,like: fs.s3a.secret.key + * @param value the value of secret key + * @return CarbonWriterBuilder object + */ + public CarbonReaderBuilder setSecretKey(String key, String value); +``` + +``` + /** + * Set the secret key for S3 + * + * @param value the value of secret key + * @return CarbonWriterBuilder object + */ + public CarbonReaderBuilder setSecretKey(String value); +``` + +``` + /** + * Set the endpoint for S3 + * + * @param key the string of endpoint for different S3 type,like: fs.s3a.endpoint + * @param value the value of endpoint + * @return CarbonWriterBuilder object + */ + public CarbonReaderBuilder setEndPoint(String key, String value); +``` + +``` + /** + * Set the endpoint for S3 + * + * @param value the value of endpoint + * @return CarbonWriterBuilder object + */ + public CarbonReaderBuilder setEndPoint(String value); +``` + +``` + /** + * Build CarbonReader + * + * @param <T> + * @return CarbonReader + * @throws IOException + * @throws InterruptedException + */ + public <T> CarbonReader<T> build(); +``` +### Class org.apache.carbondata.sdk.file.CarbonSchemaReader +``` + /** + * Read schema file and return the schema + * + * @param schemaFilePath complete path including schema file name + * @return schema object + * @throws IOException + */ + public static Schema readSchemaInSchemaFile(String schemaFilePath); +``` + +``` + /** + * Read carbondata file and return the schema + * + * @param dataFilePath complete path including carbondata file name + * @return Schema object + * @throws IOException + */ + public static Schema readSchemaInDataFile(String dataFilePath); +``` + +``` + /** + * Read carbonindex file and return the schema + * + * @param indexFilePath complete path including index file name + * @return schema object + * @throws IOException + */ + public static Schema readSchemaInIndexFile(String indexFilePath); +``` + +### Class org.apache.carbondata.sdk.file.Schema +``` + /** + * construct a schema with fields + * @param fields + */ + public Schema(Field[] fields); +``` + +``` + /** + * construct a schema with List<ColumnSchema> + * + * @param columnSchemaList column schema list + */ + public Schema(List<ColumnSchema> columnSchemaList); +``` + +``` + /** + * Create a Schema using JSON string, for example: + * [ + * {"name":"string"}, + * {"age":"int"} + * ] + * @param json specified as string + * @return Schema + */ + public static Schema parseJson(String json); +``` + +``` + /** + * Sort the schema order as original order + * + * @return Schema object + */ + public Schema asOriginOrder(); +``` + +### Class org.apache.carbondata.sdk.file.Field +``` + /** + * Field Constructor + * @param name name of the field + * @param type datatype of field, specified in strings. + */ + public Field(String name, String type); +``` + +``` + /** + * Construct Field from ColumnSchema + * + * @param columnSchema ColumnSchema, Store the information about the column meta data + */ + public Field(ColumnSchema columnSchema); +``` + +Find S3 example code at [SDKS3Example](https://github.com/apache/carbondata/blob/master/examples/spark2/src/main/java/org/apache/carbondata/examples/sdk/SDKS3Example.java) in the CarbonData repo. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/markdown/sdk-writer-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/sdk-writer-guide.md b/src/site/markdown/sdk-writer-guide.md deleted file mode 100644 index 9878b71..0000000 --- a/src/site/markdown/sdk-writer-guide.md +++ /dev/null @@ -1,359 +0,0 @@ -# SDK Writer Guide -In the carbon jars package, there exist a carbondata-store-sdk-x.x.x-SNAPSHOT.jar. -This SDK writer, writes carbondata file and carbonindex file at a given path. -External client can make use of this writer to convert other format data or live data to create carbondata and index files. -These SDK writer output contains just a carbondata and carbonindex files. No metadata folder will be present. - -## Quick example - -### Example with csv format - -```java - import java.io.IOException; - - import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException; - import org.apache.carbondata.core.metadata.datatype.DataTypes; - import org.apache.carbondata.sdk.file.CarbonWriter; - import org.apache.carbondata.sdk.file.CarbonWriterBuilder; - import org.apache.carbondata.sdk.file.Field; - import org.apache.carbondata.sdk.file.Schema; - - public class TestSdk { - - public static void main(String[] args) throws IOException, InvalidLoadOptionException { - testSdkWriter(); - } - - public static void testSdkWriter() throws IOException, InvalidLoadOptionException { - String path = "/home/root1/Documents/ab/temp"; - - Field[] fields = new Field[2]; - fields[0] = new Field("name", DataTypes.STRING); - fields[1] = new Field("age", DataTypes.INT); - - Schema schema = new Schema(fields); - - CarbonWriterBuilder builder = CarbonWriter.builder().withSchema(schema).outputPath(path); - - CarbonWriter writer = builder.buildWriterForCSVInput(); - - int rows = 5; - for (int i = 0; i < rows; i++) { - writer.write(new String[] { "robot" + (i % 10), String.valueOf(i) }); - } - writer.close(); - } - } -``` - -### Example with Avro format -```java -import java.io.IOException; - -import org.apache.carbondata.common.exceptions.sql.InvalidLoadOptionException; -import org.apache.carbondata.core.metadata.datatype.DataTypes; -import org.apache.carbondata.sdk.file.AvroCarbonWriter; -import org.apache.carbondata.sdk.file.CarbonWriter; -import org.apache.carbondata.sdk.file.Field; - -import org.apache.avro.generic.GenericData; -import org.apache.commons.lang.CharEncoding; - -import tech.allegro.schema.json2avro.converter.JsonAvroConverter; - -public class TestSdkAvro { - - public static void main(String[] args) throws IOException, InvalidLoadOptionException { - testSdkWriter(); - } - - - public static void testSdkWriter() throws IOException, InvalidLoadOptionException { - String path = "./AvroCarbonWriterSuiteWriteFiles"; - // Avro schema - String avroSchema = - "{" + - " \"type\" : \"record\"," + - " \"name\" : \"Acme\"," + - " \"fields\" : [" - + "{ \"name\" : \"fname\", \"type\" : \"string\" }," - + "{ \"name\" : \"age\", \"type\" : \"int\" }]" + - "}"; - - String json = "{\"fname\":\"bob\", \"age\":10}"; - - // conversion to GenericData.Record - JsonAvroConverter converter = new JsonAvroConverter(); - GenericData.Record record = converter.convertToGenericDataRecord( - json.getBytes(CharEncoding.UTF_8), new org.apache.avro.Schema.Parser().parse(avroSchema)); - - // prepare carbon schema from avro schema - org.apache.carbondata.sdk.file.Schema carbonSchema = - AvroCarbonWriter.getCarbonSchemaFromAvroSchema(avroSchema); - - try { - CarbonWriter writer = CarbonWriter.builder() - .withSchema(carbonSchema) - .outputPath(path) - .buildWriterForAvroInput(); - - for (int i = 0; i < 100; i++) { - writer.write(record); - } - writer.close(); - } catch (Exception e) { - e.printStackTrace(); - } - } -} -``` - -## Datatypes Mapping -Each of SQL data types are mapped into data types of SDK. Following are the mapping: - -| SQL DataTypes | Mapped SDK DataTypes | -|---------------|----------------------| -| BOOLEAN | DataTypes.BOOLEAN | -| SMALLINT | DataTypes.SHORT | -| INTEGER | DataTypes.INT | -| BIGINT | DataTypes.LONG | -| DOUBLE | DataTypes.DOUBLE | -| VARCHAR | DataTypes.STRING | -| DATE | DataTypes.DATE | -| TIMESTAMP | DataTypes.TIMESTAMP | -| STRING | DataTypes.STRING | -| DECIMAL | DataTypes.createDecimalType(precision, scale) | - - -## API List - -### Class org.apache.carbondata.sdk.file.CarbonWriterBuilder -``` -/** -* prepares the builder with the schema provided -* @param schema is instance of Schema -* This method must be called when building CarbonWriterBuilder -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withSchema(Schema schema); -``` - -``` -/** -* Sets the output path of the writer builder -* @param path is the absolute path where output files are written -* This method must be called when building CarbonWriterBuilder -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder outputPath(String path); -``` - -``` -/** -* If set false, writes the carbondata and carbonindex files in a flat folder structure -* @param isTransactionalTable is a boolelan value -* if set to false, then writes the carbondata and carbonindex files -* in a flat folder structure. -* if set to true, then writes the carbondata and carbonindex files -* in segment folder structure.. -* By default set to false. -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder isTransactionalTable(boolean isTransactionalTable); -``` - -``` -/** -* to set the timestamp in the carbondata and carbonindex index files -* @param UUID is a timestamp to be used in the carbondata and carbonindex index files. -* By default set to zero. -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder uniqueIdentifier(long UUID); -``` - -``` -/** -* To set the carbondata file size in MB between 1MB-2048MB -* @param blockSize is size in MB between 1MB to 2048 MB -* default value is 1024 MB -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withBlockSize(int blockSize); -``` - -``` -/** -* To set the blocklet size of carbondata file -* @param blockletSize is blocklet size in MB -* default value is 64 MB -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withBlockletSize(int blockletSize); -``` - -``` -/** -* sets the list of columns that needs to be in sorted order -* @param sortColumns is a string array of columns that needs to be sorted. -* If it is null or by default all dimensions are selected for sorting -* If it is empty array, no columns are sorted -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder sortBy(String[] sortColumns); -``` - -``` -/** -* If set, create a schema file in metadata folder. -* @param persist is a boolean value, If set to true, creates a schema file in metadata folder. -* By default set to false. will not create metadata folder -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder persistSchemaFile(boolean persist); -``` - -``` -/** -* sets the taskNo for the writer. SDKs concurrently running -* will set taskNo in order to avoid conflicts in file's name during write. -* @param taskNo is the TaskNo user wants to specify. -* by default it is system time in nano seconds. -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder taskNo(String taskNo); -``` - -``` -/** -* To support the load options for sdk writer -* @param options key,value pair of load options. -* supported keys values are -* a. bad_records_logger_enable -- true (write into separate logs), false -* b. bad_records_action -- FAIL, FORCE, IGNORE, REDIRECT -* c. bad_record_path -- path -* d. dateformat -- same as JAVA SimpleDateFormat -* e. timestampformat -- same as JAVA SimpleDateFormat -* f. complex_delimiter_level_1 -- value to Split the complexTypeData -* g. complex_delimiter_level_2 -- value to Split the nested complexTypeData -* h. quotechar -* i. escapechar -* -* Default values are as follows. -* -* a. bad_records_logger_enable -- "false" -* b. bad_records_action -- "FAIL" -* c. bad_record_path -- "" -* d. dateformat -- "" , uses from carbon.properties file -* e. timestampformat -- "", uses from carbon.properties file -* f. complex_delimiter_level_1 -- "$" -* g. complex_delimiter_level_2 -- ":" -* h. quotechar -- "\"" -* i. escapechar -- "\\" -* -* @return updated CarbonWriterBuilder -*/ -public CarbonWriterBuilder withLoadOptions(Map<String, String> options); -``` - -``` -/** -* Build a {@link CarbonWriter}, which accepts row in CSV format object -* @return CSVCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException -*/ -public CarbonWriter buildWriterForCSVInput() throws IOException, InvalidLoadOptionException; -``` - -``` -/** -* Build a {@link CarbonWriter}, which accepts Avro format object -* @return AvroCarbonWriter -* @throws IOException -* @throws InvalidLoadOptionException -*/ -public CarbonWriter buildWriterForAvroInput() throws IOException, InvalidLoadOptionException; -``` - -### Class org.apache.carbondata.sdk.file.CarbonWriter -``` -/** -* Write an object to the file, the format of the object depends on the implementation -* If AvroCarbonWriter, object is of type org.apache.avro.generic.GenericData.Record -* If CSVCarbonWriter, object is of type String[] -* Note: This API is not thread safe -* @param object -* @throws IOException -*/ -public abstract void write(Object object) throws IOException; -``` - -``` -/** -* Flush and close the writer -*/ -public abstract void close() throws IOException; -``` - -``` -/** -* Create a {@link CarbonWriterBuilder} to build a {@link CarbonWriter} -*/ -public static CarbonWriterBuilder builder() { -return new CarbonWriterBuilder(); -} -``` - -### Class org.apache.carbondata.sdk.file.Field -``` -/** -* Field Constructor -* @param name name of the field -* @param type datatype of field, specified in strings. -*/ -public Field(String name, String type); -``` - -``` -/** -* Field constructor -* @param name name of the field -* @param type datatype of the field of class DataType -*/ -public Field(String name, DataType type); -``` - -### Class org.apache.carbondata.sdk.file.Schema - -``` -/** -* construct a schema with fields -* @param fields -*/ -public Schema(Field[] fields); -``` - -``` -/** -* Create a Schema using JSON string, for example: -* [ -* {"name":"string"}, -* {"age":"int"} -* ] -* @param json specified as string -* @return Schema -*/ -public static Schema parseJson(String json); -``` - -### Class org.apache.carbondata.sdk.file.AvroCarbonWriter -``` -/** -* converts avro schema to carbon schema, required by carbonWriter -* -* @param avroSchemaString json formatted avro schema as string -* @return carbon sdk schema -*/ -public static org.apache.carbondata.sdk.file.Schema getCarbonSchemaFromAvroSchema(String avroSchemaString); -``` \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/markdown/streaming-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/streaming-guide.md b/src/site/markdown/streaming-guide.md index 3ea2881..a9b174f 100644 --- a/src/site/markdown/streaming-guide.md +++ b/src/site/markdown/streaming-guide.md @@ -27,11 +27,11 @@ Start spark-shell in new terminal, type :paste, then copy and run the following import org.apache.spark.sql.{CarbonEnv, SparkSession} import org.apache.spark.sql.CarbonSession._ import org.apache.spark.sql.streaming.{ProcessingTime, StreamingQuery} - import org.apache.carbondata.core.util.path.CarbonStorePath - + import org.apache.carbondata.core.util.path.CarbonTablePath + val warehouse = new File("./warehouse").getCanonicalPath val metastore = new File("./metastore").getCanonicalPath - + val spark = SparkSession .builder() .master("local") @@ -54,8 +54,8 @@ Start spark-shell in new terminal, type :paste, then copy and run the following | TBLPROPERTIES('streaming'='true')""".stripMargin) val carbonTable = CarbonEnv.getCarbonTable(Some("default"), "carbon_table")(spark) - val tablePath = CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier) - + val tablePath = carbonTable.getTablePath + // batch load var qry: StreamingQuery = null val readSocketDF = spark.readStream @@ -68,7 +68,7 @@ Start spark-shell in new terminal, type :paste, then copy and run the following qry = readSocketDF.writeStream .format("carbondata") .trigger(ProcessingTime("5 seconds")) - .option("checkpointLocation", tablePath.getStreamingCheckpointDir) + .option("checkpointLocation", CarbonTablePath.getStreamingCheckpointDir(tablePath)) .option("dbName", "default") .option("tableName", "carbon_table") .start() http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/markdown/supported-data-types-in-carbondata.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/supported-data-types-in-carbondata.md b/src/site/markdown/supported-data-types-in-carbondata.md index 6c21508..7260afe 100644 --- a/src/site/markdown/supported-data-types-in-carbondata.md +++ b/src/site/markdown/supported-data-types-in-carbondata.md @@ -38,6 +38,8 @@ * Complex Types * arrays: ARRAY``<data_type>`` * structs: STRUCT``<col_name : data_type COMMENT col_comment, ...>`` + + **NOTE**: Only 2 level complex type schema is supported for now. * Other Types * BOOLEAN \ No newline at end of file http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/markdown/timeseries-datamap-guide.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/timeseries-datamap-guide.md b/src/site/markdown/timeseries-datamap-guide.md index 7847312..bea5286 100644 --- a/src/site/markdown/timeseries-datamap-guide.md +++ b/src/site/markdown/timeseries-datamap-guide.md @@ -1,12 +1,12 @@ # CarbonData Timeseries DataMap -* [Timeseries DataMap](#timeseries-datamap-intoduction-(alpha-feature-in-1.3.0)) +* [Timeseries DataMap Introduction](#timeseries-datamap-intoduction) * [Compaction](#compacting-pre-aggregate-tables) * [Data Management](#data-management-with-pre-aggregate-tables) -## Timeseries DataMap Intoduction (Alpha feature in 1.3.0) -Timeseries DataMap a pre-aggregate table implementation based on 'preaggregate' DataMap. -Difference is that Timerseries DataMap has built-in understanding of time hierarchy and +## Timeseries DataMap Introduction (Alpha feature in 1.3.0) +Timeseries DataMap a pre-aggregate table implementation based on 'pre-aggregate' DataMap. +Difference is that Timeseries DataMap has built-in understanding of time hierarchy and levels: year, month, day, hour, minute, so that it supports automatic roll-up in time dimension for query. http://git-wip-us.apache.org/repos/asf/carbondata-site/blob/2941aaca/src/site/pdf.xml ---------------------------------------------------------------------- diff --git a/src/site/pdf.xml b/src/site/pdf.xml index e8449a1..5fc3341 100644 --- a/src/site/pdf.xml +++ b/src/site/pdf.xml @@ -16,8 +16,10 @@ <item name="Installation" ref='installation-guide.md'/> <item name="Configuring CarbonData" ref='configuration-parameters.md'/> <item name="Streaming Guide" ref='streaming-guide.md'/> - <item name="SDK Writer Guide" ref='sdk-writer-guide.md'/> - <item name="DataMap Developer Guide" ref='datamap-developer-guide.md'/> + <item name="SDK Guide" ref='sdk-guide.md'/> + <item name="DataMap Developer Guide" ref='datamap-developer-guide.md'/> + <item name="CarbonData BloomFilter DataMap (Alpha feature in 1.4.0)" ref='bloomfilter-datamap-guide.md'/> + <item name="CarbonData Lucene DataMap (Alpha feature in 1.4.0)" ref='lucene-datamap-guide.md'/> <item name="CarbonData Pre-aggregate DataMap" ref='preaggregate-datamap-guide.md'/> <item name="CarbonData Timeseries DataMap" ref='timeseries-datamap-guide.md'/> <item name="FAQs" ref='faq.md'/> @@ -30,7 +32,7 @@ <companyLogo>../../src/site/projectLogo/ApacheLogo.png</companyLogo> <projectLogo>../../src/site/projectLogo/CarbonDataLogo.png</projectLogo> <coverTitle>Apache CarbonData</coverTitle> - <coverSubTitle>Ver 1.3.1 </coverSubTitle> + <coverSubTitle>Ver 1.4.0 </coverSubTitle> <coverType>Documentation</coverType> <projectName>Apache CarbonData</projectName> <companyName>The Apache Software Foundation</companyName>
