[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user vandana7 closed the pull request at: https://github.com/apache/carbondata/pull/1205 ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130924110 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. + +* LOCAL_SORT : The sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. + +* GLOBAL_SORT : The sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. + +* NO_SORT : Feasible if we want to load our data in unsorted manner. --- End diff -- not we --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130924023 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. + +* LOCAL_SORT : The sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. + +* GLOBAL_SORT : The sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. --- End diff -- not per task, it is sorting in whole cluster --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130923792 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. --- End diff -- mention this is sort based on memory size, and carbon will create one index for each batch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130923366 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. + +* LOCAL_SORT : The sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. + +* GLOBAL_SORT : The sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. + +* NO_SORT : Feasible if we want to load our data in unsorted manner. --- End diff -- Introduce this first --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user zzcclp commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130792549 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. + +* LOCAL_SORT : The sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. + +* GLOBAL_SORT : The sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. + +* NO_SORT : Feasible if we want to load our data in unsorted manner. + +For BATCH_SORT: + +``` +OPTIONS ('SORT_SCOPE'='BATCH_SORT') +``` + +You can also specify the sort size option for sort scope. + +``` +OPTIONS('SORT_SCOPE'='BATCH_SORT', 'batch_sort_size_inmb'='7') +``` + +Note : + +* batch_sort_size_inmb : Size of data in MB to be processed in batch. By default it is the 45 percent size of sort.inmemory.size.inmb(Memory size in MB available for in-memory sort). + +For GLOBAL_SORT : --- End diff -- Sorry for my mistake, now when sort_scope='GLOBAL_SORT', single_pass can be 'true', I have raised a pr to remove this restriction of code (PR-1224). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user zzcclp commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130581766 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. + +* LOCAL_SORT : The sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. + +* GLOBAL_SORT : The sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. + +* NO_SORT : Feasible if we want to load our data in unsorted manner. + +For BATCH_SORT: + +``` +OPTIONS ('SORT_SCOPE'='BATCH_SORT') +``` + +You can also specify the sort size option for sort scope. + +``` +OPTIONS('SORT_SCOPE'='BATCH_SORT', 'batch_sort_size_inmb'='7') +``` + +Note : + +* batch_sort_size_inmb : Size of data in MB to be processed in batch. By default it is the 45 percent size of sort.inmemory.size.inmb(Memory size in MB available for in-memory sort). + +For GLOBAL_SORT : --- End diff -- I mean that if SORT_SCOPE=GLOBAL_SORT,single_pass must be false --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user vandana7 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130578370 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. + +* LOCAL_SORT : The sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. + +* GLOBAL_SORT : The sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. + +* NO_SORT : Feasible if we want to load our data in unsorted manner. + +For BATCH_SORT: + +``` +OPTIONS ('SORT_SCOPE'='BATCH_SORT') +``` + +You can also specify the sort size option for sort scope. + +``` +OPTIONS('SORT_SCOPE'='BATCH_SORT', 'batch_sort_size_inmb'='7') +``` + +Note : + +* batch_sort_size_inmb : Size of data in MB to be processed in batch. By default it is the 45 percent size of sort.inmemory.size.inmb(Memory size in MB available for in-memory sort). + +For GLOBAL_SORT : --- End diff -- Hi, I tried to execute the LOAD query with single_pass= 'true' and sort_scope='BATCH_SORT', it successfully executed and i was able to fetch the records in sorted way syntax i used to execute load query - LOAD DATA INPATH 'hdfs://localhost:54310/uniqdata/2000_UniqData.csv' into table uniqdata OPTIONS('DELIMITER'=',' , 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='TRUE','SORT_SCOPE'='BATCH_SORT','batch_sort_size_inmb'='7'); Please let me know if i am doing anything wrong --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
Github user zzcclp commented on a diff in the pull request: https://github.com/apache/carbondata/pull/1205#discussion_r130534145 --- Diff: docs/dml-operation-on-carbondata.md --- @@ -149,6 +149,50 @@ You can use the following options to load data: * If this option is set to TRUE, then high.cardinality.identify.enable property will be disabled during data load. +- **SORT_SCOPE:** This property can have four possible values : + +* BATCH_SORT : The sorting scope is smaller and more index tree will be created,thus loading is faster but query maybe slower. + +* LOCAL_SORT : The sorting scope is bigger and one index tree per data node will be created, thus loading is slower but query is faster. + +* GLOBAL_SORT : The sorting scope is bigger and one index tree per task will be created, thus loading is slower but query is faster. + +* NO_SORT : Feasible if we want to load our data in unsorted manner. + +For BATCH_SORT: + +``` +OPTIONS ('SORT_SCOPE'='BATCH_SORT') +``` + +You can also specify the sort size option for sort scope. + +``` +OPTIONS('SORT_SCOPE'='BATCH_SORT', 'batch_sort_size_inmb'='7') +``` + +Note : + +* batch_sort_size_inmb : Size of data in MB to be processed in batch. By default it is the 45 percent size of sort.inmemory.size.inmb(Memory size in MB available for in-memory sort). + +For GLOBAL_SORT : --- End diff -- Suggestion: add below note: `'SINGLE_PASS' must be false.` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] carbondata pull request #1205: [CARBONDATA-1086] updated configuration-param...
GitHub user vandana7 opened a pull request: https://github.com/apache/carbondata/pull/1205 [CARBONDATA-1086] updated configuration-parameters.md and dml-operation-on-carbondata for SORT_SCOPE You can merge this pull request into a Git repository by running: $ git pull https://github.com/vandana7/incubator-carbondata sort_scope_update Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/1205.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1205 commit 31f237e9c46247b60f84917da3c0368a5114531b Author: vandana Date: 2017-07-28T07:07:07Z updated configuration-parameters.md and dml-operation-on-carbondata.md for sort_scope feature --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---