[GitHub] incubator-carbondata pull request #161: [CARBONDATA-246] compaction is wrong...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/161 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #161: [CARBONDATA-246] compaction is wrong...
Github user ravikiran23 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79290314 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -280,6 +277,25 @@ class CarbonMergerRDD[K, V]( taskInfoList.add(new TableTaskInfo(entry._1, entry._2).asInstanceOf[Distributable]) ) } + +// prepare the details required to extract the segment properties using last segment. + --- End diff -- fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #161: [CARBONDATA-246] compaction is wrong...
Github user ravikiran23 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79290299 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -259,6 +253,9 @@ class CarbonMergerRDD[K, V]( ) ) + // keep on assigning till last one is reached. + blocksOfLastSegment = blocksOfOneSegment.asJava --- End diff -- fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #161: [CARBONDATA-246] compaction is wrong...
Github user ravikiran23 commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79290197 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -102,6 +102,11 @@ class CarbonMergerRDD[K, V]( var dataloadStatus = CarbonCommonConstants.STORE_LOADSTATUS_FAILURE val carbonSparkPartition = theSplit.asInstanceOf[CarbonSparkPartition] +// get destination segment properties as sent from driver which is of last segment. + +val segmentProperties = new SegmentProperties(carbonMergerMapping.columnSchemaList.asJava, --- End diff -- fixed --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #161: [CARBONDATA-246] compaction is wrong...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79289289 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -102,6 +102,11 @@ class CarbonMergerRDD[K, V]( var dataloadStatus = CarbonCommonConstants.STORE_LOADSTATUS_FAILURE val carbonSparkPartition = theSplit.asInstanceOf[CarbonSparkPartition] +// get destination segment properties as sent from driver which is of last segment. + +val segmentProperties = new SegmentProperties(carbonMergerMapping.columnSchemaList.asJava, --- End diff -- rename it to maxSegmentcolumnSchemaList and maxSegmentColCardinality. Write comment where it is declared --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #161: [CARBONDATA-246] compaction is wrong...
Github user gvramana commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/161#discussion_r79289428 --- Diff: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonMergerRDD.scala --- @@ -259,6 +253,9 @@ class CarbonMergerRDD[K, V]( ) ) + // keep on assigning till last one is reached. + blocksOfLastSegment = blocksOfOneSegment.asJava --- End diff -- add null check and size check --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #161: [CARBONDATA-246] compaction is wrong...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/161#discussion_r79282917
--- Diff:
integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/Compactor.scala
---
@@ -69,7 +69,9 @@ object Compactor {
schemaName,
factTableName,
validSegments,
-
carbonTable.getAbsoluteTableIdentifier.getCarbonTableIdentifier.getTableId
+
carbonTable.getAbsoluteTableIdentifier.getCarbonTableIdentifier.getTableId,
+ colCardinality = Array[Int](0),
--- End diff --
Instead of creating an empty list and reassigning it you can create a
reference of Array[Int] type like
var colCardinality: Array[Int] = null
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
