[GitHub] CarbonDataQA commented on issue #3100: [CARBONDATA-3269] Fix ArrayIndexOutOfBoundsException of Range_Column when using KryoSerializer
CarbonDataQA commented on issue #3100: [CARBONDATA-3269] Fix ArrayIndexOutOfBoundsException of Range_Column when using KryoSerializer URL: https://github.com/apache/carbondata/pull/3100#issuecomment-457486525 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10695/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (CARBONDATA-3271) CarboData provide python SDK
[ https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3271: Description: Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData For pyspark, they used py4j for python invoke java code:  https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf Please refer: # https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals # https://issues.apache.org/jira/browse/SPARK-3789 was: Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData For pyspark, they used py4j for python invoke java code:  https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf Please refer: # [1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals # [2] https://issues.apache.org/jira/browse/SPARK-3789 > CarboData provide python SDK > > > Key: CARBONDATA-3271 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3271 > Project: CarbonData > Issue Type: New Feature >Affects Versions: 1.5.1 >Reporter: xubo245 >Priority: Major > > Many users use python to install their project. It's not easy for them to use > carbon by Java/Scala/C++. And Spark also provide python SDK for users. So > it's better to provide python SDK for CarbonData > For pyspark, they used py4j for python invoke java code: >  > https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf > Please refer: > # https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals > # https://issues.apache.org/jira/browse/SPARK-3789 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3271) CarboData provide python SDK
[ https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3271: Description: Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData For pyspark, they used py4j for python invoke java code:  https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf Please refer: # [1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals # [2] https://issues.apache.org/jira/browse/SPARK-3789 was: Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData > CarboData provide python SDK > > > Key: CARBONDATA-3271 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3271 > Project: CarbonData > Issue Type: New Feature >Affects Versions: 1.5.1 >Reporter: xubo245 >Priority: Major > > Many users use python to install their project. It's not easy for them to use > carbon by Java/Scala/C++. And Spark also provide python SDK for users. So > it's better to provide python SDK for CarbonData > For pyspark, they used py4j for python invoke java code: >  > https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf > Please refer: > # [1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals > # [2] https://issues.apache.org/jira/browse/SPARK-3789 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (CARBONDATA-3271) CarboData provide python SDK
[ https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 updated CARBONDATA-3271: Description: Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData For pyspark, they used py4j for python invoke java code:  https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf Please refer: # [1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals # [2] https://issues.apache.org/jira/browse/SPARK-3789 was: Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData For pyspark, they used py4j for python invoke java code:  https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf Please refer: # [1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals # [2] https://issues.apache.org/jira/browse/SPARK-3789 > CarboData provide python SDK > > > Key: CARBONDATA-3271 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3271 > Project: CarbonData > Issue Type: New Feature >Affects Versions: 1.5.1 >Reporter: xubo245 >Priority: Major > > Many users use python to install their project. It's not easy for them to use > carbon by Java/Scala/C++. And Spark also provide python SDK for users. So > it's better to provide python SDK for CarbonData > For pyspark, they used py4j for python invoke java code: >  > https://issues.apache.org/jira/secure/attachment/12752618/PyGraphX_design_doc.pdf > Please refer: > # [1] https://cwiki.apache.org/confluence/display/SPARK/PySpark+Internals > # [2] https://issues.apache.org/jira/browse/SPARK-3789 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-3271) CarboData provide python SDK
xubo245 created CARBONDATA-3271: --- Summary: CarboData provide python SDK Key: CARBONDATA-3271 URL: https://issues.apache.org/jira/browse/CARBONDATA-3271 Project: CarbonData Issue Type: New Feature Affects Versions: 1.5.1 Reporter: xubo245 Many users use python to install their project. It's not easy for them to use carbon by Java/Scala/C++. And Spark also provide python SDK for users. So it's better to provide python SDK for CarbonData -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] CarbonDataQA commented on issue #3101: [CARBONDATA-3270] MV support groupby columns don't need be existed in the projection
CarbonDataQA commented on issue #3101: [CARBONDATA-3270] MV support groupby columns don't need be existed in the projection URL: https://github.com/apache/carbondata/pull/3101#issuecomment-457482396 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2438/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] qiuchenjian opened a new pull request #3101: [CARBONDATA-3270] MV support groupby columns don't need be existed in the projection
qiuchenjian opened a new pull request #3101: [CARBONDATA-3270] MV support
groupby columns don't need be existed in the projection
URL: https://github.com/apache/carbondata/pull/3101
【Problem】
this sql throws UnsupportedOperationException("Group by columns must be
present in project columns")
`create table mv_groupby_main(name string,height int,age int) stored by
'carbondata'
create datamap mv_groupby_main_mv using 'mv' as select sum(height) from
mv_groupby_main group by age`
[JIRA-CARBONDATA-2533](https://issues.apache.org/jira/browse/CARBONDATA-2533)
JIRA-2533's modification is not suitable
【Sulution】
modify the MV SQL logic plan to add the groupby columns to projection
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (CARBONDATA-3270) MV support that group by columns doesn't need be existed in the projection
Chenjian Qiu created CARBONDATA-3270:
Summary: MV support that group by columns doesn't need be existed
in the projection
Key: CARBONDATA-3270
URL: https://issues.apache.org/jira/browse/CARBONDATA-3270
Project: CarbonData
Issue Type: Bug
Reporter: Chenjian Qiu
MV support that group columns doesn't need be existed in the projection when
create MV
[JIRA-CARBONDATA-2533|https://issues.apache.org/jira/browse/CARBONDATA-2533]
this jira's modification is not suitable
this sql throws UnsupportedOperationException("Group by columns must be present
in project columns")
{code:java}
create table mv_groupby_main(name string,height int,age int) stored by
'carbondata'
create datamap mv_groupby_main_mv using 'mv' as select sum(height) from
mv_groupby_main group by age
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ravipesala commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto
ravipesala commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto URL: https://github.com/apache/carbondata/pull/3099#issuecomment-457476871 @manishnalla1994 Please add test case for varchar This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto
CarbonDataQA commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto URL: https://github.com/apache/carbondata/pull/3099#issuecomment-457472997 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2664/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3100: [CARBONDATA-3269] Fix ArrayIndexOutOfBoundsException of Range_Column when using KryoSerializer
CarbonDataQA commented on issue #3100: [CARBONDATA-3269] Fix ArrayIndexOutOfBoundsException of Range_Column when using KryoSerializer URL: https://github.com/apache/carbondata/pull/3100#issuecomment-457472929 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2437/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto
CarbonDataQA commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto URL: https://github.com/apache/carbondata/pull/3099#issuecomment-457472907 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10694/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] QiangCai opened a new pull request #3100: [CARBONDATA-3269] Fix ArrayIndexOutOfBoundsException of Range_Column when using KryoSerializer
QiangCai opened a new pull request #3100: [CARBONDATA-3269] Fix ArrayIndexOutOfBoundsException of Range_Column when using KryoSerializer URL: https://github.com/apache/carbondata/pull/3100 Fix ArrayIndexOutOfBoundsException of Range_Column when using KryoSerializer Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (CARBONDATA-3269) Range_column throwing ArrayIndexOutOfBoundsException when using KryoSerializer
[ https://issues.apache.org/jira/browse/CARBONDATA-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QiangCai updated CARBONDATA-3269: - Description: Reproduce: For range_column feature,When we set "spark.serializer" to "org.apache.spark.serializer.KryoSerializer", data loading will throw ArrayIndexOutOfBoundsException. Excpetion: 2019-01-25 13:00:19 ERROR DataLoadProcessorStepOnSpark$:367 - Data Loading failed for table carbon_range_column4 java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-01-25 13:00:19 ERROR TaskContextImpl:91 - Error in TaskFailureListener org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException: Data Loading failed for table carbon_range_column4 at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.org$apache$carbondata$spark$load$DataLoadProcessorStepOnSpark$$wrapException(DataLoadProcessorStepOnSpark.scala:368) at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:215) at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:210) at org.apache.spark.TaskContext$$anon$2.onTaskFailure(TaskContext.scala:144) at org.apache.spark.TaskContextImpl$$anonfun$markTaskFailed$1.apply(TaskContextImpl.scala:107) at org.apache.spark.TaskContextImpl$$anonfun$markTaskFailed$1.apply(TaskContextImpl.scala:107) at org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130) at org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128) at org.apache.spark.TaskContextImpl.markTaskFailed(TaskContextImpl.scala:106) at org.apache.spark.scheduler.Task.run(Task.scala:113) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:108) ... 4 more was: 2019-01-25 13:00:19 ERROR DataLoadProcessorStepOnSpark$:367 - Data Loading failed for table carbon_range_column4 java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-01-25 13:00:19 ERROR TaskContextImpl:91 - Error in TaskFailureListener org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException: Data Loading failed for table carbon_range_column4 at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.org$apache$carbondata$spark$load$DataLoadProcessorStepOnSpark$$wrapException(DataLoadProcessorStepOnSpark.scala:368) at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:215) at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:210) at org.apache.spark.TaskContext$$anon$2.onTaskFailure(TaskContext.scala:144) at org.apache.spark.TaskContextImpl$$anonfun$markTaskFailed$1.apply(TaskContextImpl.scala:107) at
[jira] [Created] (CARBONDATA-3269) Range_column throwing ArrayIndexOutOfBoundsException when using KryoSerializer
QiangCai created CARBONDATA-3269: Summary: Range_column throwing ArrayIndexOutOfBoundsException when using KryoSerializer Key: CARBONDATA-3269 URL: https://issues.apache.org/jira/browse/CARBONDATA-3269 Project: CarbonData Issue Type: Bug Reporter: QiangCai 2019-01-25 13:00:19 ERROR DataLoadProcessorStepOnSpark$:367 - Data Loading failed for table carbon_range_column4 java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2019-01-25 13:00:19 ERROR TaskContextImpl:91 - Error in TaskFailureListener org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException: Data Loading failed for table carbon_range_column4 at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.org$apache$carbondata$spark$load$DataLoadProcessorStepOnSpark$$wrapException(DataLoadProcessorStepOnSpark.scala:368) at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:215) at org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:210) at org.apache.spark.TaskContext$$anon$2.onTaskFailure(TaskContext.scala:144) at org.apache.spark.TaskContextImpl$$anonfun$markTaskFailed$1.apply(TaskContextImpl.scala:107) at org.apache.spark.TaskContextImpl$$anonfun$markTaskFailed$1.apply(TaskContextImpl.scala:107) at org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130) at org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128) at org.apache.spark.TaskContextImpl.markTaskFailed(TaskContextImpl.scala:106) at org.apache.spark.scheduler.Task.run(Task.scala:113) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:108) ... 4 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (CARBONDATA-3269) Range_column throwing ArrayIndexOutOfBoundsException when using KryoSerializer
[ https://issues.apache.org/jira/browse/CARBONDATA-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] QiangCai reassigned CARBONDATA-3269: Assignee: QiangCai > Range_column throwing ArrayIndexOutOfBoundsException when using KryoSerializer > -- > > Key: CARBONDATA-3269 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3269 > Project: CarbonData > Issue Type: Bug >Reporter: QiangCai >Assignee: QiangCai >Priority: Critical > > 2019-01-25 13:00:19 ERROR DataLoadProcessorStepOnSpark$:367 - Data Loading > failed for table carbon_range_column4 > java.lang.ArrayIndexOutOfBoundsException: 5 > at > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:108) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2019-01-25 13:00:19 ERROR TaskContextImpl:91 - Error in TaskFailureListener > org.apache.carbondata.processing.loading.exception.CarbonDataLoadingException: > Data Loading failed for table carbon_range_column4 > at > org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$.org$apache$carbondata$spark$load$DataLoadProcessorStepOnSpark$$wrapException(DataLoadProcessorStepOnSpark.scala:368) > at > org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:215) > at > org.apache.carbondata.spark.load.DataLoadProcessorStepOnSpark$$anonfun$convertFunc$3.apply(DataLoadProcessorStepOnSpark.scala:210) > at org.apache.spark.TaskContext$$anon$2.onTaskFailure(TaskContext.scala:144) > at > org.apache.spark.TaskContextImpl$$anonfun$markTaskFailed$1.apply(TaskContextImpl.scala:107) > at > org.apache.spark.TaskContextImpl$$anonfun$markTaskFailed$1.apply(TaskContextImpl.scala:107) > at > org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:130) > at > org.apache.spark.TaskContextImpl$$anonfun$invokeListeners$1.apply(TaskContextImpl.scala:128) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at > org.apache.spark.TaskContextImpl.invokeListeners(TaskContextImpl.scala:128) > at org.apache.spark.TaskContextImpl.markTaskFailed(TaskContextImpl.scala:106) > at org.apache.spark.scheduler.Task.run(Task.scala:113) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ArrayIndexOutOfBoundsException: 5 > at > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:108) > ... 4 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] CarbonDataQA commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto
CarbonDataQA commented on issue #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto URL: https://github.com/apache/carbondata/pull/3099#issuecomment-457462945 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2436/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] manishnalla1994 opened a new pull request #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto
manishnalla1994 opened a new pull request #3099: [CARBONDATA-3268] Fix for query on Varchar Columns showing Null in Presto URL: https://github.com/apache/carbondata/pull/3099 Problem: Select query on Varchar columns shows null in Presto as it was going to ObjectStreamReader instead of SliceStreamReader. Solution: Handled the scenario by adding a check for Varchar while creating StreamReader. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [x] Any interfaces changed? -> NO - [x] Any backward compatibility impacted? -> NO - [x] Document update required? -> NO - [x] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (CARBONDATA-3268) Query on Varchar showing as Null in Presto
MANISH NALLA created CARBONDATA-3268: Summary: Query on Varchar showing as Null in Presto Key: CARBONDATA-3268 URL: https://issues.apache.org/jira/browse/CARBONDATA-3268 Project: CarbonData Issue Type: Bug Reporter: MANISH NALLA Assignee: MANISH NALLA -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #3054: [CARBONDATA-3232] Add example and doc for alluxio integration
asfgit closed pull request #3054: [CARBONDATA-3232] Add example and doc for alluxio integration URL: https://github.com/apache/carbondata/pull/3054 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename
CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#issuecomment-457236763 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2663/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename
CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#issuecomment-457236509 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10693/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename
CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#issuecomment-457216268 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2435/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (CARBONDATA-3252) Remove unused import and optimize the import order
[ https://issues.apache.org/jira/browse/CARBONDATA-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xubo245 closed CARBONDATA-3252. --- Resolution: Fixed > Remove unused import and optimize the import order > --- > > Key: CARBONDATA-3252 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3252 > Project: CarbonData > Issue Type: Bug >Reporter: xubo245 >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Remove unused import and fix some spell error > * org.apache.carbondata.spark.testsuite.badrecordloger.BadRecordLoggerTest: > > remove CarbonLoadOptionConstants in line 27 > * > org.apache.carbondata.spark.testsuite.directdictionary.TimestampNoDictionaryColumnTestCase: > remove line 23 and 26 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename
CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#issuecomment-457212221 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2662/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #3087: [CARBONDATA-3252] Remove unused import
asfgit closed pull request #3087: [CARBONDATA-3252] Remove unused import URL: https://github.com/apache/carbondata/pull/3087 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] zzcclp commented on issue #3087: [CARBONDATA-3252] Remove unused import
zzcclp commented on issue #3087: [CARBONDATA-3252] Remove unused import URL: https://github.com/apache/carbondata/pull/3087#issuecomment-457208783 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename
CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#issuecomment-457201757 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2434/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] NamanRastogi commented on issue #3094: [CARBONDATA-3264] Added SORT_SCOPE in ALTER TABLE SET
NamanRastogi commented on issue #3094: [CARBONDATA-3264] Added SORT_SCOPE in ALTER TABLE SET URL: https://github.com/apache/carbondata/pull/3094#issuecomment-457201607 @kumarvishal09 Those are not unwanted changes in document because H4 should have been used for sub-points of ALTER TABLE, but H5 was used. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename
CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#issuecomment-457207447 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10692/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3079: [WIP][CARBONDATA-3255] Support binary data type
CarbonDataQA commented on issue #3079: [WIP][CARBONDATA-3255] Support binary data type URL: https://github.com/apache/carbondata/pull/3079#issuecomment-457190745 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2659/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3079: [WIP][CARBONDATA-3255] Support binary data type
CarbonDataQA commented on issue #3079: [WIP][CARBONDATA-3255] Support binary data type URL: https://github.com/apache/carbondata/pull/3079#issuecomment-457190387 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10690/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename
CarbonDataQA commented on issue #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#issuecomment-457187810 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2433/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster.
asfgit closed pull request #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster. URL: https://github.com/apache/carbondata/pull/3089 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3079: [WIP][CARBONDATA-3255] Support binary data type
CarbonDataQA commented on issue #3079: [WIP][CARBONDATA-3255] Support binary data type URL: https://github.com/apache/carbondata/pull/3079#issuecomment-457174832 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2432/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 commented on a change in pull request #3087: [CARBONDATA-3252] Remove unused import
xubo245 commented on a change in pull request #3087: [CARBONDATA-3252] Remove unused import URL: https://github.com/apache/carbondata/pull/3087#discussion_r249422065 ## File path: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/badrecordloger/BadRecordLoggerTest.scala ## @@ -36,7 +35,6 @@ import org.apache.carbondata.core.datastore.impl.FileFactory * Review comment: Can you remove line 34 and line 35? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 commented on issue #3087: [CARBONDATA-3252] Remove unused import
xubo245 commented on issue #3087: [CARBONDATA-3252] Remove unused import URL: https://github.com/apache/carbondata/pull/3087#issuecomment-457173423 LGTM, thanks for your contribution! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] xubo245 commented on issue #3037: [CARBONDATA-3190] Open example module code style check and fix some style errors
xubo245 commented on issue #3037: [CARBONDATA-3190] Open example module code style check and fix some style errors URL: https://github.com/apache/carbondata/pull/3037#issuecomment-457172667 @sraghunandan @chenliang613 @KanakaKumar @jackylk @QiangCai @zzcclp @ravipesala Please review it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (CARBONDATA-3265) Memory Leak and Low Query Performance Issues in Range Partition
[ https://issues.apache.org/jira/browse/CARBONDATA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kunal Kapoor resolved CARBONDATA-3265. -- Resolution: Fixed Fix Version/s: 1.5.2 > Memory Leak and Low Query Performance Issues in Range Partition > --- > > Key: CARBONDATA-3265 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3265 > Project: CarbonData > Issue Type: Bug >Reporter: Shivam Goyal >Priority: Major > Fix For: 1.5.2 > > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #3095: [CARBONDATA-3265] Fixed memory leak in Range Sort
asfgit closed pull request #3095: [CARBONDATA-3265] Fixed memory leak in Range Sort URL: https://github.com/apache/carbondata/pull/3095 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kunal642 commented on issue #3095: [CARBONDATA-3265] Fixed memory leak in Range Sort
kunal642 commented on issue #3095: [CARBONDATA-3265] Fixed memory leak in Range Sort URL: https://github.com/apache/carbondata/pull/3095#issuecomment-457167024 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster.
CarbonDataQA commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster. URL: https://github.com/apache/carbondata/pull/3089#issuecomment-457146029 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10689/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster.
CarbonDataQA commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster. URL: https://github.com/apache/carbondata/pull/3089#issuecomment-457146066 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2658/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shivamasn commented on a change in pull request #3095: [CARBONDATA-3265] Fixed memory leak in Range Sort
shivamasn commented on a change in pull request #3095: [CARBONDATA-3265] Fixed
memory leak in Range Sort
URL: https://github.com/apache/carbondata/pull/3095#discussion_r250535291
##
File path:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##
@@ -127,9 +130,32 @@ object DataLoadProcessBuilderOnSpark {
}
// 4. Write
-sc.runJob(sortRDD, (context: TaskContext, rows: Iterator[CarbonRow]) =>
+sc.runJob(sortRDD, (context: TaskContext, rows: Iterator[CarbonRow]) => {
+ TaskContext.get.addTaskCompletionListener(_ =>
ThreadLocalSessionInfo.unsetAll())
+ val carbonSessionInfo: CarbonSessionInfo = {
+var info = ThreadLocalSessionInfo.getCarbonSessionInfo
+if (info == null || null == info.getSessionParams) {
Review comment:
This piece of code is not required, so I have removed it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] kumarvishal09 commented on issue #3094: [CARBONDATA-3264] Added SORT_SCOPE in ALTER TABLE SET
kumarvishal09 commented on issue #3094: [CARBONDATA-3264] Added SORT_SCOPE in ALTER TABLE SET URL: https://github.com/apache/carbondata/pull/3094#issuecomment-457137901 @NamanRastogi There are some unwanted changes add in this PR can u please check and revert the same This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] kumarvishal09 commented on a change in pull request #3098: [CARBONDATA-3235] Fixed Alter Table Rename
kumarvishal09 commented on a change in pull request #3098: [CARBONDATA-3235] Fixed Alter Table Rename URL: https://github.com/apache/carbondata/pull/3098#discussion_r250533327 ## File path: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/schema/CarbonAlterTableRenameCommand.scala ## @@ -61,6 +61,8 @@ private[sql] case class CarbonAlterTableRenameCommand( } val oldTableName = oldTableIdentifier.table.toLowerCase val newTableName = newTableIdentifier.table.toLowerCase +val oldIdentifier = TableIdentifier(oldTableName, Some(oldDatabaseName)) Review comment: what the purpose of creating oldIdentifier and newIdentifier again?? when already both identifiers are present https://github.com/apache/carbondata/pull/3098/files#diff-9fb2c5a8b63af8d61563a154cdfc8a3cR46 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster.
CarbonDataQA commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster. URL: https://github.com/apache/carbondata/pull/3089#issuecomment-457126109 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2431/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (CARBONDATA-3267) Data loading is failing with OOM using range sort
[ https://issues.apache.org/jira/browse/CARBONDATA-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-3267. - Resolution: Fixed Fix Version/s: 1.5.2 > Data loading is failing with OOM using range sort > - > > Key: CARBONDATA-3267 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3267 > Project: CarbonData > Issue Type: Bug >Reporter: kumar vishal >Assignee: kumar vishal >Priority: Major > Fix For: 1.5.2 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > h3. Problem: > Range sort is failing with OOM. > h3. Root cause: > This is because UnsafeSortStorageMemory is not able to control the off heap > memory because of this when huge data is loaded it OOM exception is coming > fron UnsafeMemoryAllocator.allocate. > h3. Solution: > Control Sort Storage memory. After sorting the rows if memory is available > then only add sorted records to sort storage memory otherwise write to disk -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] ravipesala commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster.
ravipesala commented on issue #3089: [HOTFIX] presto carbon doesn't work with Hadoop conf in cluster. URL: https://github.com/apache/carbondata/pull/3089#issuecomment-457122737 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit closed pull request #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue
asfgit closed pull request #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue URL: https://github.com/apache/carbondata/pull/3096 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] ravipesala commented on issue #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue
ravipesala commented on issue #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue URL: https://github.com/apache/carbondata/pull/3096#issuecomment-457121652 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue
CarbonDataQA commented on issue #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue URL: https://github.com/apache/carbondata/pull/3096#issuecomment-457112957 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2657/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] CarbonDataQA commented on issue #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue
CarbonDataQA commented on issue #3096: [CARBONDATA-3267]Fixed Range Sort OOM Issue URL: https://github.com/apache/carbondata/pull/3096#issuecomment-457108671 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10688/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
