[
https://issues.apache.org/jira/browse/CARBONDATA-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
QiangCai updated CARBONDATA-3347:
---------------------------------
Description:
*Background*
Now SORT_COLUMNS can’t be modified after the table is created. If we want to
modify SORT_COLUMNS on this table, we need to create a new table and migrate
data. If the data is huge, the migration will take a long time and even impact
the user business.
SORT_SCOPE in table properties can be modified now. And we can specify new
SORT_SCOPE during data loading. Carbon index file will mark whether this
segment is sorted or not. So the different segments maybe have different
SORT_SCOPE.
*Mo**tivation*
After the table is created, the user can adjust SORT_SCOPE/SORT_COLUMNS
according to their business. History segments will still use old
SORT_SCOPE/SORT_COLUMNS, but the user also can resort old segments one by one
if need.
But we still suggest the user give a proper SORT_SCOPE/SORT_COLUMNS when they
create the table because the modification will take many resources to resort
data of old segments.
please check design doc for more detail.
[^sort_columns modification.pdf]
[^sort_columns modification_v2.pdf]
was:
*Background*
Now SORT_COLUMNS can’t be modified after the table is created. If we want to
modify SORT_COLUMNS on this table, we need to create a new table and migrate
data. If the data is huge, the migration will take a long time and even impact
the user business.
SORT_SCOPE in table properties can be modified now. And we can specify new
SORT_SCOPE during data loading. Carbon index file will mark whether this
segment is sorted or not. So the different segments maybe have different
SORT_SCOPE.
*Mo**tivation*
After the table is created, the user can adjust SORT_SCOPE/SORT_COLUMNS
according to their business. History segments will still use old
SORT_SCOPE/SORT_COLUMNS, but the user also can resort old segments one by one
if need.
But we still suggest the user give a proper SORT_SCOPE/SORT_COLUMNS when they
create the table because the modification will take many resources to resort
data of old segments.
please check design doc for more detail.
[^sort_columns modification.pdf]
> support SORT_COLUMNS modification
> ---------------------------------
>
> Key: CARBONDATA-3347
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3347
> Project: CarbonData
> Issue Type: New Feature
> Components: spark-integration
> Reporter: QiangCai
> Assignee: QiangCai
> Priority: Major
> Attachments: sort_columns modification.pdf, sort_columns
> modification_v2.pdf
>
>
> *Background*
> Now SORT_COLUMNS can’t be modified after the table is created. If we want to
> modify SORT_COLUMNS on this table, we need to create a new table and migrate
> data. If the data is huge, the migration will take a long time and even
> impact the user business.
> SORT_SCOPE in table properties can be modified now. And we can specify new
> SORT_SCOPE during data loading. Carbon index file will mark whether this
> segment is sorted or not. So the different segments maybe have different
> SORT_SCOPE.
> *Mo**tivation*
> After the table is created, the user can adjust SORT_SCOPE/SORT_COLUMNS
> according to their business. History segments will still use old
> SORT_SCOPE/SORT_COLUMNS, but the user also can resort old segments one by one
> if need.
> But we still suggest the user give a proper SORT_SCOPE/SORT_COLUMNS when they
> create the table because the modification will take many resources to resort
> data of old segments.
>
> please check design doc for more detail.
> [^sort_columns modification.pdf]
> [^sort_columns modification_v2.pdf]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
