Partition features of Spark:
1. Creating table with partition
CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name1 col_type1 [COMMENT col_comment1], ...)]
USING datasource
[OPTIONS (key1=val1, key2=val2, ...)]
[PARTITIONED BY (col_name1, col_name2, ...)]
[TBLPROPERTIES (key1=val1, key2=val2, ...)]
[AS select_statement]
2. Load data
Static Partition
LOAD DATA LOCAL INPATH '${env:HOME}/staticinput.txt'
INTO TABLE partitioned_user
PARTITION (country = 'US', state = 'CA')
INSERT OVERWRITE TABLE partitioned_user
PARTITION (country = 'US', state = 'AL')
SELECT * FROM another_user au
WHERE au.country = 'US' AND au.state = 'AL';
Dynamic Partition
LOAD DATA LOCAL INPATH '${env:HOME}/staticinput.txt'
INTO TABLE partitioned_user
PARTITION (country, state)
INSERT OVERWRITE TABLE partitioned_user
PARTITION (country, state)
SELECT * FROM another_user;
3. Drop, show partitions
SHOW PARTITIONS [db_name.]table_name
ALTER TABLE table_name DROP [IF EXISTS] (PARTITION part_spec, ...)
4. Updating the partitions
ALTER TABLE table_name PARTITION part_spec RENAME TO PARTITION part_spec
Currently, carbon supports the partitions which is custom implemented by
carbon. So if community users want to use the features which are available
in spark and hive in carbondata then there is a compatibility problem
arrives. And also carbondata does not have built in dynamic partition.
To use the partition feature of spark we should comply with the interfaces
available in spark while loading and reading the data.
Approach 1 :
Comply with pure spark datasource API and implement standard interfaces for
reading and writing of data at a file level.Just like how parquet and ORC
got implemented in spark carbondata also can be implemented in the same way.
To support it we need to implement a FileFormat interface for reading and
writing the data at filelevel, not table level. For reading, we should
implement CarbonFileInputFormat(Read data at the file level) and implement
CarbonOutputFormat(Writes data per partition.)
Pros :
1.It is the clean interface to use on spark, all features of spark can be
worked without any impact.
2.Upgrading from new versions of spark is straightforward and simple.
Cons:
All Carbondata features such as IUD, compaction, Alter table and data
management like show segments, delete segments etc cannot work.
Approach 2:
Improve and expand the current in-house partition features which already
exist in carbondata. Add all the missing features like dynamic partition
and comply the syntax of loading data to partitions.
Pros :
All current features of carbondata works without much impact.
Cons:
Current partition implementation does not comply to spark partition so need
to spend a lot of effort to implement it.
Approach 3:
It is the hybrid approach of 1st approach. Basically, write the data using
FileFormat and CarbonOutputFormat interfaces. So all the partition
information would be added to hive automatically since we are creating the
datasource table. We make sure that the current folder structure does not
change while writing the data.Here we maintain the mapping file inside
segment folder for mapping between the partition and carbonindex file. And
while reading we first get the partition information from the hive and do
the pruning and based on the pruned partitions read the partition mapping
file to get the carbonindex for querying.
Here we will not support the current carbondata partition feature but we
support spark partition features.
Pros:
1. Support the standard interface for loading data. So features like
partition and bucketing automatically supported.
2. All standard SQL syntax works fine with this approach.
3. All current features of carbon also work fine.
Cons:
1.Existing partition feature cannot work.
2.Minor impact on features like compaction, IUD, clean files because of
maintaining the partition mapping file.
--
Thanks & Regards,
Ravindra
