Hi, I prefer the approach 3. If we use approach 3, hive, presto integration can also do partition pruning for carbon, right?
Regards, Jacky > 在 2017年11月21日，下午10:56，Ravindra Pesala <[email protected]> 写道： > > Partition features of Spark: > > 1. Creating table with partition > CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db_name.]table_name > [(col_name1 col_type1 [COMMENT col_comment1], ...)] > USING datasource > [OPTIONS (key1=val1, key2=val2, ...)] > [PARTITIONED BY (col_name1, col_name2, ...)] > [TBLPROPERTIES (key1=val1, key2=val2, ...)] > [AS select_statement] > > 2. Load data > Static Partition > > LOAD DATA LOCAL INPATH '${env:HOME}/staticinput.txt' > INTO TABLE partitioned_user > PARTITION (country = 'US', state = 'CA') > > INSERT OVERWRITE TABLE partitioned_user > PARTITION (country = 'US', state = 'AL') > SELECT * FROM another_user au > WHERE au.country = 'US' AND au.state = 'AL'; > > Dynamic Partition > > LOAD DATA LOCAL INPATH '${env:HOME}/staticinput.txt' > INTO TABLE partitioned_user > PARTITION (country, state) > > INSERT OVERWRITE TABLE partitioned_user > PARTITION (country, state) > SELECT * FROM another_user; > > 3. Drop, show partitions > SHOW PARTITIONS [db_name.]table_name > ALTER TABLE table_name DROP [IF EXISTS] (PARTITION part_spec, ...) > > 4. Updating the partitions > ALTER TABLE table_name PARTITION part_spec RENAME TO PARTITION part_spec > > > Currently, carbon supports the partitions which is custom implemented by > carbon. So if community users want to use the features which are available > in spark and hive in carbondata then there is a compatibility problem > arrives. And also carbondata does not have built in dynamic partition. > To use the partition feature of spark we should comply with the interfaces > available in spark while loading and reading the data. > > Approach 1 : > Comply with pure spark datasource API and implement standard interfaces for > reading and writing of data at a file level.Just like how parquet and ORC > got implemented in spark carbondata also can be implemented in the same way. > To support it we need to implement a FileFormat interface for reading and > writing the data at filelevel, not table level. For reading, we should > implement CarbonFileInputFormat(Read data at the file level) and implement > CarbonOutputFormat(Writes data per partition.) > Pros : > 1.It is the clean interface to use on spark, all features of spark can be > worked without any impact. > 2.Upgrading from new versions of spark is straightforward and simple. > Cons: > All Carbondata features such as IUD, compaction, Alter table and data > management like show segments, delete segments etc cannot work. > > Approach 2: > Improve and expand the current in-house partition features which already > exist in carbondata. Add all the missing features like dynamic partition > and comply the syntax of loading data to partitions. > Pros : > All current features of carbondata works without much impact. > Cons: > Current partition implementation does not comply to spark partition so need > to spend a lot of effort to implement it. > > Approach 3: > It is the hybrid approach of 1st approach. Basically, write the data using > FileFormat and CarbonOutputFormat interfaces. So all the partition > information would be added to hive automatically since we are creating the > datasource table. We make sure that the current folder structure does not > change while writing the data.Here we maintain the mapping file inside > segment folder for mapping between the partition and carbonindex file. And > while reading we first get the partition information from the hive and do > the pruning and based on the pruned partitions read the partition mapping > file to get the carbonindex for querying. > Here we will not support the current carbondata partition feature but we > support spark partition features. > Pros: > 1. Support the standard interface for loading data. So features like > partition and bucketing automatically supported. > 2. All standard SQL syntax works fine with this approach. > 3. All current features of carbon also work fine. > Cons: > 1.Existing partition feature cannot work. > 2.Minor impact on features like compaction, IUD, clean files because of > maintaining the partition mapping file. > > -- > Thanks & Regards, > Ravindra
