[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3769: [WIP][Perf] Upgrade zstd-jni version to supportReusableBuffer
CarbonDataQA1 commented on pull request #3769: URL: https://github.com/apache/carbondata/pull/3769#issuecomment-629018940 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3036/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3769: [WIP][Perf] Upgrade zstd-jni version to supportReusableBuffer
CarbonDataQA1 commented on pull request #3769: URL: https://github.com/apache/carbondata/pull/3769#issuecomment-629018314 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1317/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3769: [WIP][Perf] Upgrade zstd-jni version to supportReusableBuffer
CarbonDataQA1 commented on pull request #3769: URL: https://github.com/apache/carbondata/pull/3769#issuecomment-628665254 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3035/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3769: [WIP][Perf] Upgrade zstd-jni version to supportReusableBuffer
CarbonDataQA1 commented on pull request #3769: URL: https://github.com/apache/carbondata/pull/3769#issuecomment-628663695 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1316/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai opened a new pull request #3769: [WIP][Perf] Upgrade zstd-jni version to supportReusableBuffer
QiangCai opened a new pull request #3769: URL: https://github.com/apache/carbondata/pull/3769 ### Why is this PR needed? ### What changes were proposed in this PR? ### Does this PR introduce any user interface change? - No - Yes. (please explain the change and update document) ### Is any new testcase added? - No - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3767: [CARBONDATA-3791] update pycarbon document
CarbonDataQA1 commented on pull request #3767: URL: https://github.com/apache/carbondata/pull/3767#issuecomment-628559916 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3034/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3767: [CARBONDATA-3791] update pycarbon document
CarbonDataQA1 commented on pull request #3767: URL: https://github.com/apache/carbondata/pull/3767#issuecomment-628558598 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1315/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3766: [WIP] Support RuntimeFitler for inner equi-join
CarbonDataQA1 commented on pull request #3766: URL: https://github.com/apache/carbondata/pull/3766#issuecomment-628515015 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1313/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3766: [WIP] Support RuntimeFitler for inner equi-join
CarbonDataQA1 commented on pull request #3766: URL: https://github.com/apache/carbondata/pull/3766#issuecomment-628514313 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3032/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3767: [CARBONDATA-3791] update pycarbon document
ajantha-bhat commented on a change in pull request #3767: URL: https://github.com/apache/carbondata/pull/3767#discussion_r424971480 ## File path: python/setup.py ## @@ -44,7 +44,6 @@ 'six>=1.5.0', 'torchvision>=0.2.1', 'tensorflow>=1.4.0', -'jnius>=1.1.0', Review comment: ok. I reverted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3767: [CARBONDATA-3791] update pycarbon document
ajantha-bhat commented on a change in pull request #3767: URL: https://github.com/apache/carbondata/pull/3767#discussion_r424949095 ## File path: python/setup.py ## @@ -44,7 +44,6 @@ 'six>=1.5.0', 'torchvision>=0.2.1', 'tensorflow>=1.4.0', -'jnius>=1.1.0', Review comment: pyjnius doesn't work in windows ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3767: [CARBONDATA-3791] update pycarbon document
ajantha-bhat commented on a change in pull request #3767: URL: https://github.com/apache/carbondata/pull/3767#discussion_r424948739 ## File path: python/README.md ## @@ -6,7 +6,7 @@ PyCarbon provides python API for integrating CarbonData with AI framework like $ git clone https://github.com/apache/carbondata.git -$ cd python/pycarbon Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] xubo245 commented on a change in pull request #3767: [CARBONDATA-3791] update pycarbon document
xubo245 commented on a change in pull request #3767: URL: https://github.com/apache/carbondata/pull/3767#discussion_r424944474 ## File path: python/README.md ## @@ -6,7 +6,7 @@ PyCarbon provides python API for integrating CarbonData with AI framework like $ git clone https://github.com/apache/carbondata.git -$ cd python/pycarbon Review comment: suggesion: add pycarbon fold to line 17 and other place This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] xubo245 commented on a change in pull request #3767: [CARBONDATA-3791] update pycarbon document
xubo245 commented on a change in pull request #3767: URL: https://github.com/apache/carbondata/pull/3767#discussion_r424941724 ## File path: python/setup.py ## @@ -44,7 +44,6 @@ 'six>=1.5.0', 'torchvision>=0.2.1', 'tensorflow>=1.4.0', -'jnius>=1.1.0', Review comment: windows need jnius, please check it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3764: [CARBONDATA-3820] Support GlobalSort in the CDC
ajantha-bhat commented on a change in pull request #3764: URL: https://github.com/apache/carbondata/pull/3764#discussion_r424911515 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetCommand.scala ## @@ -267,7 +267,8 @@ case class CarbonMergeDataSetCommand( StructField(status_on_mergeds, IntegerType))) val factory = new SparkCarbonFileFormat().prepareWrite(sparkSession, job, - carbonTable.getTableInfo.getFactTable.getTableProperties.asScala.toMap, schema) +carbonTable.getTableInfo.getFactTable.getTableProperties.asScala Review comment: or we can add new table property instead of using the source table property to sort the tupleid, status_on_mergeds if it can improve query performance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] kunal642 closed pull request #3695: [WIP] partition optimization
kunal642 closed pull request #3695: URL: https://github.com/apache/carbondata/pull/3695 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3764: [CARBONDATA-3820] Support GlobalSort in the CDC
ajantha-bhat commented on a change in pull request #3764: URL: https://github.com/apache/carbondata/pull/3764#discussion_r424905868 ## File path: integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetCommand.scala ## @@ -267,7 +267,8 @@ case class CarbonMergeDataSetCommand( StructField(status_on_mergeds, IntegerType))) val factory = new SparkCarbonFileFormat().prepareWrite(sparkSession, job, - carbonTable.getTableInfo.getFactTable.getTableProperties.asScala.toMap, schema) +carbonTable.getTableInfo.getFactTable.getTableProperties.asScala Review comment: I don't see any use of table properties for this intermediate (dummy table) of [tupleid, status_on_mergeds] May be we can pass an **empty map** now for table property? @marchpure , @ravipesala This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3764: [CARBONDATA-3820] Support GlobalSort in the CDC
ajantha-bhat commented on a change in pull request #3764:
URL: https://github.com/apache/carbondata/pull/3764#discussion_r424899528
##
File path:
examples/spark/src/main/scala/org/apache/carbondata/examples/CDCExample.scala
##
@@ -141,6 +141,8 @@ object CDCExample {
.write
.format("carbondata")
.option("tableName", "target")
+.option("sort_scope", "global_sort")
Review comment:
I see the same problem for local sort also,
Also no sort with sort_columns also has this problem.
**So change PR name.** to "Fix CDC failure when sort columns present in
source dataframe"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3764: [CARBONDATA-3820] Support GlobalSort in the CDC
ajantha-bhat commented on a change in pull request #3764:
URL: https://github.com/apache/carbondata/pull/3764#discussion_r424899596
##
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/merge/MergeTestCase.scala
##
@@ -142,6 +159,22 @@ class MergeTestCase extends QueryTest with
BeforeAndAfterAll {
checkAnswer(sql("select count(*) from order where state = 2"), Seq(Row(2)))
}
+ test("test basic merge into the globalsort table") {
Review comment:
Add a test case for local sort also
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3764: [CARBONDATA-3820] Support GlobalSort in the CDC
ajantha-bhat commented on a change in pull request #3764:
URL: https://github.com/apache/carbondata/pull/3764#discussion_r424899528
##
File path:
examples/spark/src/main/scala/org/apache/carbondata/examples/CDCExample.scala
##
@@ -141,6 +141,8 @@ object CDCExample {
.write
.format("carbondata")
.option("tableName", "target")
+.option("sort_scope", "global_sort")
Review comment:
I see the same problem for local sort also,
**So change PR name.**
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
