[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387] Support Partition with MV datamap & Show DataMap Status
Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387]
Support Partition with MV datamap & Show DataMap Status
URL: https://github.com/apache/carbondata/pull/3216#discussion_r285902856
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonDataMapShowCommand.scala
##
@@ -92,7 +100,44 @@ case class CarbonDataMapShowCommand(tableIdentifier:
Option[TableIdentifier])
.map(p => s"'${ p._1 }'='${ p._2 }'").toSeq
.sorted.mkString(", ")
}
-Row(s.getDataMapName, s.getProviderName, table, dmPropertieStr)
+ // Get datamap status and sync information details
+ var dataMapStatus = "NA"
+ var syncInfo: String = "NA"
+ if (!s.getProviderName.equalsIgnoreCase(
+DataMapClassProvider.PREAGGREGATE.getShortName) &&
!s.getProviderName.equalsIgnoreCase(
+DataMapClassProvider.TIMESERIES.getShortName)) {
+if (DataMapStatusManager.getEnabledDataMapStatusDetails
+ .exists(_.getDataMapName.equalsIgnoreCase(s.getDataMapName))) {
+ dataMapStatus = DataMapStatus.ENABLED.name()
+} else {
+ dataMapStatus = DataMapStatus.DISABLED.name()
+}
+val loadMetadataDetails = SegmentStatusManager
+ .readLoadMetadata(CarbonTablePath
+.getMetadataPath(s.getRelationIdentifier.getTablePath))
+if (!s.isIndexDataMap && loadMetadataDetails.nonEmpty) {
+ breakable({
+for (i <- loadMetadataDetails.length - 1 to 0 by -1) {
+ if
(loadMetadataDetails(i).getSegmentStatus.equals(SegmentStatus.SUCCESS)) {
+val segmentMaps =
+
DataMapSegmentStatusUtil.getSegmentMap(loadMetadataDetails(i).getExtraInfo)
+val syncInfoMap = new util.HashMap[String, String]()
+val iterator = segmentMaps.entrySet().iterator()
+while (iterator.hasNext) {
+ val entry = iterator.next()
+ syncInfoMap.put(entry.getKey,
entry.getValue.get(entry.getValue.size() - 1))
+}
+syncInfoMap
+ .put(CarbonCommonConstants.LOAD_SYNC_TIME,
+loadMetadataDetails(i).getLoadEndTime.toString)
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387] Support Partition with MV datamap & Show DataMap Status
Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387]
Support Partition with MV datamap & Show DataMap Status
URL: https://github.com/apache/carbondata/pull/3216#discussion_r285902923
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/datamap/CarbonCreateDataMapCommand.scala
##
@@ -101,6 +101,28 @@ case class CarbonCreateDataMapCommand(
s" with provider ${dataMapSchema.getProviderName}")
}
+if (null != mainTable) {
+ if (mainTable.isChildTable || mainTable.isChildDataMap) {
+throw new MalformedDataMapCommandException(
+ "Cannot create DataMap on child table " + mainTable.getDatabaseName
+ "." +
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387] Support Partition with MV datamap & Show DataMap Status
Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387]
Support Partition with MV datamap & Show DataMap Status
URL: https://github.com/apache/carbondata/pull/3216#discussion_r285709865
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/datamap/MVHelper.scala
##
@@ -694,5 +731,28 @@ object MVHelper {
rewrittenPlan
}
}
+
+ private def getUpdatedOutputList(outputList: Seq[NamedExpression],
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387] Support Partition with MV datamap & Show DataMap Status
Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387]
Support Partition with MV datamap & Show DataMap Status
URL: https://github.com/apache/carbondata/pull/3216#discussion_r285709756
##
File path:
core/src/main/java/org/apache/carbondata/core/datamap/DataMapProvider.java
##
@@ -264,23 +264,52 @@ private boolean
getSpecificSegmentsTobeLoaded(Map> segmentM
} else {
for (RelationIdentifier relationIdentifier : relationIdentifiers) {
List dataMapTableSegmentList = new ArrayList<>();
+// Get all segments for parent relationIdentifier
+List mainTableSegmentList =
+DataMapUtil.getMainTableValidSegmentList(relationIdentifier);
+boolean ifTableStatusUpdateRequired = false;
for (LoadMetadataDetails loadMetaDetail : listOfLoadFolderDetails) {
if (loadMetaDetail.getSegmentStatus() == SegmentStatus.SUCCESS
|| loadMetaDetail.getSegmentStatus() ==
SegmentStatus.INSERT_IN_PROGRESS) {
Map> segmentMaps =
DataMapSegmentStatusUtil.getSegmentMap(loadMetaDetail.getExtraInfo());
+for (String segmentId : mainTableSegmentList) {
+ // In case if dataMap segment(0) is mapped to mainTable
segments{0,1,2} and if
+ // {0,1,2} segments of mainTable are compacted to 0.1. Then,
+ // on next rebuild/load to dataMap, no need to load segment(0.1)
again. Update the
+ // segmentMapping of dataMap segment from {0,1,2} to {0.1}
+ if (!checkIfSegmentsToBeReloaded(relationIdentifier,
segmentMaps.get(
+ relationIdentifier.getDatabaseName() +
CarbonCommonConstants.POINT
+ + relationIdentifier.getTableName()), segmentId)) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387] Support Partition with MV datamap & Show DataMap Status
Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387]
Support Partition with MV datamap & Show DataMap Status
URL: https://github.com/apache/carbondata/pull/3216#discussion_r285709807
##
File path:
datamap/mv/core/src/main/scala/org/apache/carbondata/mv/datamap/MVDataMapProvider.scala
##
@@ -163,9 +165,47 @@ class MVDataMapProvider(
SparkSQLUtil.execute(loadCommand, sparkSession)
} catch {
case ex: Exception =>
+ // If load to dataMap table fails, disable the dataMap and if
newLoad is still
+ // in INSERT_IN_PROGRESS state, mark for delete the newLoad and
update table status file
DataMapStatusManager.disableDataMap(dataMapSchema.getDataMapName)
LOGGER.error("Data Load failed for DataMap: ", ex)
- return false
+ val segmentStatusManager = new SegmentStatusManager(dataMapTable
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387] Support Partition with MV datamap & Show DataMap Status
Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387]
Support Partition with MV datamap & Show DataMap Status
URL: https://github.com/apache/carbondata/pull/3216#discussion_r285709714
##
File path:
core/src/main/java/org/apache/carbondata/core/datamap/DataMapProvider.java
##
@@ -264,23 +264,52 @@ private boolean
getSpecificSegmentsTobeLoaded(Map> segmentM
} else {
for (RelationIdentifier relationIdentifier : relationIdentifiers) {
List dataMapTableSegmentList = new ArrayList<>();
+// Get all segments for parent relationIdentifier
+List mainTableSegmentList =
+DataMapUtil.getMainTableValidSegmentList(relationIdentifier);
+boolean ifTableStatusUpdateRequired = false;
for (LoadMetadataDetails loadMetaDetail : listOfLoadFolderDetails) {
if (loadMetaDetail.getSegmentStatus() == SegmentStatus.SUCCESS
|| loadMetaDetail.getSegmentStatus() ==
SegmentStatus.INSERT_IN_PROGRESS) {
Map> segmentMaps =
DataMapSegmentStatusUtil.getSegmentMap(loadMetaDetail.getExtraInfo());
+for (String segmentId : mainTableSegmentList) {
+ // In case if dataMap segment(0) is mapped to mainTable
segments{0,1,2} and if
+ // {0,1,2} segments of mainTable are compacted to 0.1. Then,
+ // on next rebuild/load to dataMap, no need to load segment(0.1)
again. Update the
+ // segmentMapping of dataMap segment from {0,1,2} to {0.1}
+ if (!checkIfSegmentsToBeReloaded(relationIdentifier,
segmentMaps.get(
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387] Support Partition with MV datamap & Show DataMap Status
Indhumathi27 commented on a change in pull request #3216: [CARBONDATA-3387]
Support Partition with MV datamap & Show DataMap Status
URL: https://github.com/apache/carbondata/pull/3216#discussion_r285709509
##
File path:
datamap/mv/plan/src/main/scala/org/apache/carbondata/mv/plans/util/BirdcageOptimizer.scala
##
@@ -96,7 +96,7 @@ object BirdcageOptimizer extends RuleExecutor[LogicalPlan] {
// OptimizeIn(conf),
ConstantFolding,
ReorderAssociativeOperator,
-LikeSimplification,
+//LikeSimplification,
Review comment:
okay.. done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
