[GitHub] [carbondata] QiangCai commented on a change in pull request #3810: [CARBONDATA-3900] [CARBONDATA-3882] [CARBONDATA-3881] Fix multiple concurrent issues in table status lock and segment lock f

2020-07-16 Thread GitBox
QiangCai commented on a change in pull request #3810: URL: https://github.com/apache/carbondata/pull/3810#discussion_r456203328 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/util/SecondaryIndexUtil.scala ## @@ -440,14 +448,22 @@ object

[GitHub] [carbondata] QiangCai commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-07-16 Thread GitBox
QiangCai commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-659829122 @ajantha-bhat 1. remove CarbonData jars from your local maven repo at first. 2. build it with -o, you will find dependency error( can not find the dependency:

[GitHub] [carbondata] tianlileer closed pull request #710: [CARBONDATA-833]load data from dataframe,generater data row may be error when delimiter…

2020-07-16 Thread GitBox
tianlileer closed pull request #710: URL: https://github.com/apache/carbondata/pull/710 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[jira] [Created] (CARBONDATA-3911) NullPointerException is thrown when clean files is executed after two updates

2020-07-16 Thread Akash R Nilugal (Jira)
Akash R Nilugal created CARBONDATA-3911: --- Summary: NullPointerException is thrown when clean files is executed after two updates Key: CARBONDATA-3911 URL:

[GitHub] [carbondata] QiangCai commented on pull request #3778: [WIP] Support array with SI

2020-07-16 Thread GitBox
QiangCai commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-659830702 please describe the PR and fix the failure. This is an automated message from the Apache Git Service. To

[jira] [Created] (CARBONDATA-3910) load fails when csv file present in local and loading to cluster

2020-07-16 Thread Akash R Nilugal (Jira)
Akash R Nilugal created CARBONDATA-3910: --- Summary: load fails when csv file present in local and loading to cluster Key: CARBONDATA-3910 URL: https://issues.apache.org/jira/browse/CARBONDATA-3910

[GitHub] [carbondata] akashrn5 commented on pull request #3838: [CARBONDATA-3910]Fix load failure in cluster when csv present in local file system in case of global sort

2020-07-16 Thread GitBox
akashrn5 commented on pull request #3838: URL: https://github.com/apache/carbondata/pull/3838#issuecomment-659863917 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [carbondata] Zhangshunyu commented on pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
Zhangshunyu commented on pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#issuecomment-659810429 please check the build failure info This is an automated message from the Apache Git Service. To respond

[GitHub] [carbondata] QiangCai commented on a change in pull request #3785: [CARBONDATA-3843] Fix merge index issue in streaming table

2020-07-16 Thread GitBox
QiangCai commented on a change in pull request #3785: URL: https://github.com/apache/carbondata/pull/3785#discussion_r456179554 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/events/MergeIndexEventListener.scala ## @@ -104,73 +104,80 @@ class

[GitHub] [carbondata] Zhangshunyu commented on a change in pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
Zhangshunyu commented on a change in pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#discussion_r456193818 ## File path: processing/src/main/java/org/apache/carbondata/processing/loading/sort/unsafe/UnsafeSortDataRows.java ## @@ -200,25 +203,44 @@

[GitHub] [carbondata] Zhangshunyu commented on a change in pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
Zhangshunyu commented on a change in pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#discussion_r456193999 ## File path: processing/src/main/java/org/apache/carbondata/processing/sort/sortdata/SortParameters.java ## @@ -37,6 +40,13 @@ import

[GitHub] [carbondata] QiangCai commented on a change in pull request #3842: [CARBONDATA-3702] Clean temp index files in parallel in merge index flow

2020-07-16 Thread GitBox
QiangCai commented on a change in pull request #3842: URL: https://github.com/apache/carbondata/pull/3842#discussion_r456206506 ## File path: integration/spark/src/main/scala/org/apache/spark/rdd/CarbonMergeFilesRDD.scala ## @@ -157,21 +157,21 @@ object CarbonMergeFilesRDD {

[GitHub] [carbondata] QiangCai commented on a change in pull request #3842: [CARBONDATA-3702] Clean temp index files in parallel in merge index flow

2020-07-16 Thread GitBox
QiangCai commented on a change in pull request #3842: URL: https://github.com/apache/carbondata/pull/3842#discussion_r456206506 ## File path: integration/spark/src/main/scala/org/apache/spark/rdd/CarbonMergeFilesRDD.scala ## @@ -157,21 +157,21 @@ object CarbonMergeFilesRDD {

[jira] [Updated] (CARBONDATA-3905) When there are many segment files presto query fail

2020-07-16 Thread XiaoWen (Jira)
[ https://issues.apache.org/jira/browse/CARBONDATA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoWen updated CARBONDATA-3905: Description: test case1 insert data in: {code:java} df.writeStream.foreachBatch{ (batchDF:

[jira] [Created] (CARBONDATA-3905) When there are many segment files presto query fail

2020-07-16 Thread XiaoWen (Jira)
XiaoWen created CARBONDATA-3905: --- Summary: When there are many segment files presto query fail Key: CARBONDATA-3905 URL: https://issues.apache.org/jira/browse/CARBONDATA-3905 Project: CarbonData

[jira] [Commented] (CARBONDATA-3904) insert into data got Failed to create directory path /d

2020-07-16 Thread Kunal Kapoor (Jira)
[ https://issues.apache.org/jira/browse/CARBONDATA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17158972#comment-17158972 ] Kunal Kapoor commented on CARBONDATA-3904: -- What is the warehouse location? HDFS/S3? >

[GitHub] [carbondata] ajantha-bhat commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-07-16 Thread GitBox
ajantha-bhat commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-659307068 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [carbondata] IceMimosa opened a new pull request #3848: [CARBONDATA-3891] Fix loading data will update all segments updateDeltaEndTimestamp

2020-07-16 Thread GitBox
IceMimosa opened a new pull request #3848: URL: https://github.com/apache/carbondata/pull/3848 ### Why is this PR needed? Loading Data to the partitioned table will update all segments updateDeltaEndTimestamp,that will cause the driver to clear all segments cache when doing the

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#issuecomment-659311646 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1661/

[GitHub] [carbondata] ajantha-bhat commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-07-16 Thread GitBox
ajantha-bhat commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-659305724 @QiangCai : Developer should not manually modify pom to make it work for spark2.4. so I fixed like above. But some testcase failed to find CSV file after

[GitHub] [carbondata] ShreelekhyaG opened a new pull request #3849: [WIP] table level timestampformat

2020-07-16 Thread GitBox
ShreelekhyaG opened a new pull request #3849: URL: https://github.com/apache/carbondata/pull/3849 ### Why is this PR needed? To support timestamp format table level. ### What changes were proposed in this PR? Made the priority of timestamp format as: 1) Load command

[GitHub] [carbondata] QiangCai commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-07-16 Thread GitBox
QiangCai commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-659304203 finalName is ${artifactId}-${version} by default, this change will not impact artifactId and version. Other modules will can not find the dependency:

[GitHub] [carbondata] IceMimosa commented on pull request #3848: [CARBONDATA-3891] Fix loading data will update all segments updateDeltaEndTimestamp

2020-07-16 Thread GitBox
IceMimosa commented on pull request #3848: URL: https://github.com/apache/carbondata/pull/3848#issuecomment-659312101 reset please This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [carbondata] ajantha-bhat commented on pull request #3787: [WIP] support sort_scope for index creation

2020-07-16 Thread GitBox
ajantha-bhat commented on pull request #3787: URL: https://github.com/apache/carbondata/pull/3787#issuecomment-659312911 @QiangCai : yes, it is in WIP. SI global sort I will support from this PR This is an automated message

[GitHub] [carbondata] QiangCai commented on pull request #3787: [WIP] support sort_scope for index creation

2020-07-16 Thread GitBox
QiangCai commented on pull request #3787: URL: https://github.com/apache/carbondata/pull/3787#issuecomment-659312148 during SI loading, it should use this sort_scope. This is an automated message from the Apache Git Service.

[jira] [Created] (CARBONDATA-3906) Optimize sort performance in writting file

2020-07-16 Thread bishunli (Jira)
bishunli created CARBONDATA-3906: Summary: Optimize sort performance in writting file Key: CARBONDATA-3906 URL: https://issues.apache.org/jira/browse/CARBONDATA-3906 Project: CarbonData

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#issuecomment-659309836 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3402/

[GitHub] [carbondata] ajantha-bhat edited a comment on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-07-16 Thread GitBox
ajantha-bhat edited a comment on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-659305724 @QiangCai : Developer should not manually modify pom to make it work for spark2.4. After this PR both 2.4 and 2.5 works without any manual change and jar names

[GitHub] [carbondata] ajantha-bhat commented on pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
ajantha-bhat commented on pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#issuecomment-659307892 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [carbondata] shunlean opened a new pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
shunlean opened a new pull request #3847: URL: https://github.com/apache/carbondata/pull/3847 ### Why is this PR needed? Only after sorting temp, the write(sortTemp file) operation can run. For better performance, we want to do the writeDataToFile and SortDataRows operations in

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#issuecomment-659300018 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [carbondata] ajantha-bhat commented on pull request #3847: [CARBONDATA-3906] Optimize sort performance in writting file

2020-07-16 Thread GitBox
ajantha-bhat commented on pull request #3847: URL: https://github.com/apache/carbondata/pull/3847#issuecomment-659307713 Add to whitelist This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-659339600 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3403/

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3848: [CARBONDATA-3891] Fix loading data will update all segments updateDeltaEndTimestamp

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3848: URL: https://github.com/apache/carbondata/pull/3848#issuecomment-659376770 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1662/

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3807: [HOTFIX] Fix module problems of mv and spark with spark binary version

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3807: URL: https://github.com/apache/carbondata/pull/3807#issuecomment-659338632 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1660/

[jira] [Created] (CARBONDATA-3907) Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutionEvent and LoadTablePostExecutionEvent respectively in alt

2020-07-16 Thread Venugopal Reddy K (Jira)
Venugopal Reddy K created CARBONDATA-3907: - Summary: Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutionEvent and LoadTablePostExecutionEvent respectively in alter table add

[jira] [Created] (CARBONDATA-3908) When a carbon segment is added through the alter add segments query, then it is not accounting the added carbon segment values.

2020-07-16 Thread Prasanna Ravichandran (Jira)
Prasanna Ravichandran created CARBONDATA-3908: - Summary: When a carbon segment is added through the alter add segments query, then it is not accounting the added carbon segment values. Key: CARBONDATA-3908

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3849: [WIP] table level timestampformat

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-659392712 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3407/

[GitHub] [carbondata] VenuReddy2103 opened a new pull request #3850: [CARBONDATA-3907]Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutionEvent

2020-07-16 Thread GitBox
VenuReddy2103 opened a new pull request #3850: URL: https://github.com/apache/carbondata/pull/3850 ### Why is this PR needed? Currently we have 2 different ways of firing LoadTablePreExecutionEvent and LoadTablePostExecutionEvent. We can reuse firePreLoadEvents and firePostLoadEvents

[jira] [Updated] (CARBONDATA-3907) Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutionEvent and LoadTablePostExecutionEvent respectively in alt

2020-07-16 Thread Venugopal Reddy K (Jira)
[ https://issues.apache.org/jira/browse/CARBONDATA-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venugopal Reddy K updated CARBONDATA-3907: -- Description: *[Issue]* Currently we have 2 different ways of firing

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3848: [CARBONDATA-3891] Fix loading data will update all segments updateDeltaEndTimestamp

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3848: URL: https://github.com/apache/carbondata/pull/3848#issuecomment-659380516 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3404/

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3850: [CARBONDATA-3907]Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutionEvent

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#issuecomment-659422935 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3849: [WIP] table level timestampformat

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-659447560 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1665/

[GitHub] [carbondata] ShreelekhyaG commented on pull request #3849: [WIP] table level timestampformat

2020-07-16 Thread GitBox
ShreelekhyaG commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-659467768 retest this please This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [carbondata] Indhumathi27 opened a new pull request #3851: [WIP]Fix Global sort data load failure issue with Decimal value as NULL

2020-07-16 Thread GitBox
Indhumathi27 opened a new pull request #3851: URL: https://github.com/apache/carbondata/pull/3851 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain

[jira] [Created] (CARBONDATA-3909) Insert into select fails after insert decimal value as null and set sort scope to global sort

2020-07-16 Thread Chetan Bhat (Jira)
Chetan Bhat created CARBONDATA-3909: --- Summary: Insert into select fails after insert decimal value as null and set sort scope to global sort Key: CARBONDATA-3909 URL:

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3849: [WIP] table level timestampformat

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-659550711 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3408/

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3849: [WIP] table level timestampformat

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-659554650 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1666/

[GitHub] [carbondata] akashrn5 commented on a change in pull request #3850: [CARBONDATA-3907]Reuse firePreLoadEvents and firePostLoadEvents methods from CommonLoadUtils to trigger LoadTablePreExecutio

2020-07-16 Thread GitBox
akashrn5 commented on a change in pull request #3850: URL: https://github.com/apache/carbondata/pull/3850#discussion_r455974523 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAddLoadCommand.scala ## @@ -228,24 +228,17 @@

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3851: [WIP]Fix Global sort data load failure issue with Decimal value as NULL

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3851: URL: https://github.com/apache/carbondata/pull/3851#issuecomment-659592635 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3409/

[GitHub] [carbondata] akashrn5 commented on a change in pull request #3851: [WIP]Fix Global sort data load failure issue with Decimal value as NULL

2020-07-16 Thread GitBox
akashrn5 commented on a change in pull request #3851: URL: https://github.com/apache/carbondata/pull/3851#discussion_r455977271 ## File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestLoadDataGeneral.scala ## @@ -234,9

[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3851: [WIP]Fix Global sort data load failure issue with Decimal value as NULL

2020-07-16 Thread GitBox
CarbonDataQA1 commented on pull request #3851: URL: https://github.com/apache/carbondata/pull/3851#issuecomment-659591221 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1667/

[GitHub] [carbondata] QiangCai commented on a change in pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types

2020-07-16 Thread GitBox
QiangCai commented on a change in pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#discussion_r456167613 ## File path: integration/spark/src/test/scala/org/apache/carbondata/integration/spark/testsuite/complexType/TestArrayContainsPushDown.scala ## @@

[GitHub] [carbondata] Zhangshunyu commented on pull request #3849: [WIP] table level timestampformat

2020-07-16 Thread GitBox
Zhangshunyu commented on pull request #3849: URL: https://github.com/apache/carbondata/pull/3849#issuecomment-659774957 Greate! This is a useful feature. This is an automated message from the Apache Git Service. To respond

[GitHub] [carbondata] QiangCai commented on a change in pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types

2020-07-16 Thread GitBox
QiangCai commented on a change in pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#discussion_r456175101 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/strategy/CarbonLateDecodeStrategy.scala ## @@ -865,7 +870,33 @@

[GitHub] [carbondata] QiangCai commented on a change in pull request #3771: [CARBONDATA-3849] pushdown array_contains filter to carbon for array of primitive types

2020-07-16 Thread GitBox
QiangCai commented on a change in pull request #3771: URL: https://github.com/apache/carbondata/pull/3771#discussion_r456176045 ## File path: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java ## @@ -222,49 +228,103 @@ public