[GitHub] [carbondata] xubo245 opened a new pull request #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK
xubo245 opened a new pull request #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK URL: https://github.com/apache/carbondata/pull/3567 ### Why is this PR needed? Fix Conflicts when concurrently write data by SDK and didn't set taskNo ### What changes were proposed in this PR? set Default TaskNo when TaskNo is null ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] QiangCai commented on issue #3561: [HOTFIX] Fix INSERT STAGE footer read error
QiangCai commented on issue #3561: [HOTFIX] Fix INSERT STAGE footer read error URL: https://github.com/apache/carbondata/pull/3561#issuecomment-573292476 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] xubo245 closed pull request #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK
xubo245 closed pull request #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK URL: https://github.com/apache/carbondata/pull/3567 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] xubo245 commented on issue #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK
xubo245 commented on issue #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK URL: https://github.com/apache/carbondata/pull/3567#issuecomment-573288133 CI pass This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert. URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573288006 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1595/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert. URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573278489 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1594/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert. URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573273445 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1593/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK
CarbonDataQA1 commented on issue #3567: [CARBONDATA-3656] set Default TaskNo To Avoid Conflicts when concurrently write data by SDK URL: https://github.com/apache/carbondata/pull/3567#issuecomment-573119723 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1591/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert.
CarbonDataQA1 commented on issue #3538: [WIP] Separate Insert and load to later optimize insert. URL: https://github.com/apache/carbondata/pull/3538#issuecomment-573118720 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1592/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-573049648 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1590/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-573021802 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-573020241 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-573017141 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1587/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3571: [CARBONDATA-3659] Fix issues with alluxio without host and port.
CarbonDataQA1 commented on issue #3571: [CARBONDATA-3659] Fix issues with alluxio without host and port. URL: https://github.com/apache/carbondata/pull/3571#issuecomment-573014185 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1585/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-572990984 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180545
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala
##
@@ -562,7 +563,13 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
protected lazy val showCache: Parser[LogicalPlan] =
SHOW ~> METACACHE ~> opt(ontable) <~ opt(";") ^^ {
case table =>
-CarbonShowCacheCommand(table)
+CarbonShowCacheCommand(false, table)
+}
+
+ protected lazy val showExecutorCache: Parser[LogicalPlan] =
+SHOW ~> EXECUTOR ~> METACACHE ~> opt(ontable) <~ opt(";") ^^ {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180496
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -135,23 +193,46 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
carbonTables.flatMap {
mainTable =>
try {
-makeRows(getTableCacheFromIndexServer(mainTable)(sparkSession),
mainTable)
+val row = makeRows(getTableCacheFromIndexServer(showExecutorCache,
+ mainTable)(sparkSession), mainTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
+}
+if (res(1) == 0 && res(2) == 0 && res(3) == 0) {
+ Seq()
+} else {
+ row
+}
} catch {
case ex: UnsupportedOperationException => Seq()
}
}
-} else { Seq() }
+} else {
+ Seq()
+}
val driverRows = if (cache != null) {
carbonTables.flatMap {
carbonTable =>
try {
-makeRows(getTableCacheFromDriver(sparkSession, carbonTable),
carbonTable)
+val row = makeRows(getTableCacheFromDriver(sparkSession,
carbonTable), carbonTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180462
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -135,23 +193,46 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
carbonTables.flatMap {
mainTable =>
try {
-makeRows(getTableCacheFromIndexServer(mainTable)(sparkSession),
mainTable)
+val row = makeRows(getTableCacheFromIndexServer(showExecutorCache,
+ mainTable)(sparkSession), mainTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
+}
+if (res(1) == 0 && res(2) == 0 && res(3) == 0) {
+ Seq()
+} else {
+ row
+}
} catch {
case ex: UnsupportedOperationException => Seq()
}
}
-} else { Seq() }
+} else {
+ Seq()
+}
val driverRows = if (cache != null) {
carbonTables.flatMap {
carbonTable =>
try {
-makeRows(getTableCacheFromDriver(sparkSession, carbonTable),
carbonTable)
+val row = makeRows(getTableCacheFromDriver(sparkSession,
carbonTable), carbonTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
+}
+if (res(1) == 0 && res(2) == 0 && res(3) == 0) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180429
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -171,39 +252,33 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
.toList)
if (driverRows.nonEmpty) {
(Seq(
- Row("ALL", "ALL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER"),
- Row(currentDatabase,
-"ALL",
-driverdbIndexSize,
-driverdbDatamapSize,
-driverdbDictSize,
-"DRIVER")
+ Row("TOTAL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER")
) ++ driverRows).collect {
- case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
-Row(row(0), row(1), bytesToDisplaySize(row.getLong(2)),
- bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "DRIVER")
+ case row if row.getLong(1) != 0L || row.getLong(2) != 0L ||
row.getLong(3) != 0L =>
+Row(row(0), bytesToDisplaySize(row.getLong(1)),
+ bytesToDisplaySize(row.getLong(2)),
bytesToDisplaySize(row.getLong(3)), "DRIVER")
}
} else {
-makeEmptyCacheRows(currentDatabase)
+makeEmptyCacheRows(currentDatabase, "DRIVER")
}
} else {
- makeEmptyCacheRows(currentDatabase)
+ makeEmptyCacheRows(currentDatabase, "DRIVER")
}
-// val (serverIndexSize, serverDataMapSize) =
getAllIndexServerCacheSize
-val indexDisplayRows = if (indexServerRows.nonEmpty) {
- (Seq(
-Row("ALL", "ALL", indexAllIndexSize, indexAllDatamapSize,
indexAllDictSize, "INDEX SERVER"),
-Row(currentDatabase,
- "ALL",
- indexdbIndexSize,
- indexdbDatamapSize,
- driverdbDictSize,
- "INDEX SERVER")
- ) ++ indexServerRows).collect {
-case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
- Row(row.get(0), row.get(1), bytesToDisplaySize(row.getLong(2)),
-bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "INDEX SERVER")
+val indexDisplayRows = if (isDistributedPruningEnabled) {
+ if (indexServerRows.nonEmpty) {
+(Seq(
+ Row("TOTAL", indexAllIndexSize, indexAllDatamapSize,
indexAllDictSize, "INDEX SERVER")
+) ++ indexServerRows).collect {
+ case row if row.getLong(1) != 0L || row.getLong(2) != 0L ||
row.getLong(3) != 0L =>
+Row(row.get(0),
+ bytesToDisplaySize(row.getLong(1)),
+ bytesToDisplaySize(row.getLong(2)),
+ bytesToDisplaySize(row.getLong(3)),
+ "INDEX SERVER")
+}
+ } else {
+makeEmptyCacheRows(currentDatabase, "INDEXSERVER")
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180105
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -110,13 +137,42 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
}
}
+ def getAllExecutorCache(sparkSession: SparkSession): Seq[Row] = {
+val isDistributedPruningEnabled = CarbonProperties.getInstance()
+ .isDistributedPruningEnabled("", "")
+if (!isDistributedPruningEnabled) {
+ getAllTablesCache(sparkSession)
+}
+else {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180130
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -110,13 +137,42 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
}
}
+ def getAllExecutorCache(sparkSession: SparkSession): Seq[Row] = {
+val isDistributedPruningEnabled = CarbonProperties.getInstance()
+ .isDistributedPruningEnabled("", "")
+if (!isDistributedPruningEnabled) {
+ getAllTablesCache(sparkSession)
+}
+else {
+ // get all the executor details from the index server
+ try {
+val executorCacheValue = executeJobToGetCache(showExecutorCache,
List())
+val result = executorCacheValue.flatMap {
+ iterator =>
+Seq(Row(iterator._1, bytesToDisplaySize(iterator._3)))
+}
+result
+ }
+ catch {
+case ex: Exception =>
+ LOGGER.error("Error while getting cache from the Index Server", ex)
+ Seq()
+ }
+}
+ }
+
def getAllTablesCache(sparkSession: SparkSession): Seq[Row] = {
val currentDatabase = sparkSession.sessionState.catalog.getCurrentDatabase
val cache = CacheProvider.getInstance().getCarbonCache
val isDistributedPruningEnabled = CarbonProperties.getInstance()
.isDistributedPruningEnabled("", "")
-if (cache == null && !isDistributedPruningEnabled) {
- return makeEmptyCacheRows(currentDatabase)
+if (!isDistributedPruningEnabled) {
+ if (cache == null) {
+return makeEmptyCacheRows(currentDatabase, "DRIVER")
+ }
+ if (cache.getCurrentSize == 0) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180259
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -171,39 +252,33 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
.toList)
if (driverRows.nonEmpty) {
(Seq(
- Row("ALL", "ALL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER"),
- Row(currentDatabase,
-"ALL",
-driverdbIndexSize,
-driverdbDatamapSize,
-driverdbDictSize,
-"DRIVER")
+ Row("TOTAL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER")
) ++ driverRows).collect {
- case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
-Row(row(0), row(1), bytesToDisplaySize(row.getLong(2)),
- bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "DRIVER")
+ case row if row.getLong(1) != 0L || row.getLong(2) != 0L ||
row.getLong(3) != 0L =>
+Row(row(0), bytesToDisplaySize(row.getLong(1)),
+ bytesToDisplaySize(row.getLong(2)),
bytesToDisplaySize(row.getLong(3)), "DRIVER")
}
} else {
-makeEmptyCacheRows(currentDatabase)
+makeEmptyCacheRows(currentDatabase, "DRIVER")
}
} else {
- makeEmptyCacheRows(currentDatabase)
+ makeEmptyCacheRows(currentDatabase, "DRIVER")
}
-// val (serverIndexSize, serverDataMapSize) =
getAllIndexServerCacheSize
-val indexDisplayRows = if (indexServerRows.nonEmpty) {
- (Seq(
-Row("ALL", "ALL", indexAllIndexSize, indexAllDatamapSize,
indexAllDictSize, "INDEX SERVER"),
-Row(currentDatabase,
- "ALL",
- indexdbIndexSize,
- indexdbDatamapSize,
- driverdbDictSize,
- "INDEX SERVER")
- ) ++ indexServerRows).collect {
-case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
- Row(row.get(0), row.get(1), bytesToDisplaySize(row.getLong(2)),
-bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "INDEX SERVER")
+val indexDisplayRows = if (isDistributedPruningEnabled) {
+ if (indexServerRows.nonEmpty) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180193
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -127,6 +183,8 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
carbonTables += carbonTable
}
} catch {
+ case ex: AnalysisException =>
+LOGGER.info("Unable to access Carbon table object for table" +
tableIdent.table)
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180075
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -41,25 +42,42 @@ import org.apache.carbondata.spark.util.CarbonScalaUtil
import org.apache.carbondata.spark.util.CommonUtil.bytesToDisplaySize
-case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
+case class CarbonShowCacheCommand(showExecutorCache: Boolean,
+tableIdentifier: Option[TableIdentifier],
internalCall: Boolean = false)
extends MetadataCommand {
private lazy val cacheResult: Seq[(String, Int, Long, String)] = {
-executeJobToGetCache(List())
+executeJobToGetCache(showExecutorCache, List())
}
private val LOGGER =
LogServiceFactory.getLogService(classOf[CarbonShowCacheCommand].getName)
override def output: Seq[AttributeReference] = {
if (tableIdentifier.isEmpty) {
- Seq(
-AttributeReference("Database", StringType, nullable = false)(),
-AttributeReference("Table", StringType, nullable = false)(),
-AttributeReference("Index size", StringType, nullable = false)(),
-AttributeReference("Datamap size", StringType, nullable = false)(),
-AttributeReference("Dictionary size", StringType, nullable = false)(),
-AttributeReference("Cache Location", StringType, nullable = false)())
+ val isDistributedPruningEnabled = CarbonProperties.getInstance()
+.isDistributedPruningEnabled("", "")
+ if (showExecutorCache == false) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180004
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -74,7 +92,15 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
/**
* Assemble result for database
*/
- getAllTablesCache(sparkSession)
+ if (showExecutorCache == false) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365180029
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -41,25 +42,42 @@ import org.apache.carbondata.spark.util.CarbonScalaUtil
import org.apache.carbondata.spark.util.CommonUtil.bytesToDisplaySize
-case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
+case class CarbonShowCacheCommand(showExecutorCache: Boolean,
+tableIdentifier: Option[TableIdentifier],
internalCall: Boolean = false)
extends MetadataCommand {
private lazy val cacheResult: Seq[(String, Int, Long, String)] = {
-executeJobToGetCache(List())
+executeJobToGetCache(showExecutorCache, List())
}
private val LOGGER =
LogServiceFactory.getLogService(classOf[CarbonShowCacheCommand].getName)
override def output: Seq[AttributeReference] = {
if (tableIdentifier.isEmpty) {
- Seq(
-AttributeReference("Database", StringType, nullable = false)(),
-AttributeReference("Table", StringType, nullable = false)(),
-AttributeReference("Index size", StringType, nullable = false)(),
-AttributeReference("Datamap size", StringType, nullable = false)(),
-AttributeReference("Dictionary size", StringType, nullable = false)(),
-AttributeReference("Cache Location", StringType, nullable = false)())
+ val isDistributedPruningEnabled = CarbonProperties.getInstance()
+.isDistributedPruningEnabled("", "")
+ if (showExecutorCache == false) {
+Seq(
+ AttributeReference("Database and Table", StringType, nullable =
false)(),
+ AttributeReference("Index size", StringType, nullable = false)(),
+ AttributeReference("Datamap size", StringType, nullable = false)(),
+ AttributeReference("Dictionary size", StringType, nullable =
false)(),
+ AttributeReference("Cache Location", StringType, nullable = false)())
+ } else {
+if (!isDistributedPruningEnabled) {
+ Seq(
+AttributeReference("Database and Table", StringType, nullable =
false)(),
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365179930
##
File path:
integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
##
@@ -57,7 +57,7 @@ trait ServerInterface {
/**
* Get the cache size for the specified tables.
*/
- def showCache(tableIds: String) : Array[String]
+ def showCache(executorCache: Boolean, tableIds: String) : Array[String]
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-572988742 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1584/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365179961
##
File path:
integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
##
@@ -205,14 +205,18 @@ object IndexServer extends ServerInterface {
}
}
- override def showCache(tableId: String = ""): Array[String] = doAs {
+ override def showCache(executorCache: Boolean, tableId: String = ""):
Array[String] = doAs {
val jobgroup: String = "Show Cache " + (tableId match {
- case "" => "for all tables"
+ case "" =>
+executorCache match {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
vikramahuja1001 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365179893
##
File path:
integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedShowCacheRDD.scala
##
@@ -71,10 +75,21 @@ class DistributedShowCacheRDD(@transient private val ss:
SparkSession, tableUniq
.getTableUniqueName
} else {
dataMap.getDataMapSchema.getRelationIdentifier.getDatabaseName +
"_" + dataMap
- .getDataMapSchema.getDataMapName
+.getDataMapSchema.getDataMapName
+}
+if (executorCache == true) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] ravipesala commented on a change in pull request #3571: [CARBONDATA-3659] Fix issues with alluxio without host and port.
ravipesala commented on a change in pull request #3571: [CARBONDATA-3659] Fix
issues with alluxio without host and port.
URL: https://github.com/apache/carbondata/pull/3571#discussion_r365178265
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/filesystem/AbstractDFSCarbonFile.java
##
@@ -554,6 +555,21 @@ public short getDefaultReplication() {
return fileSystem.getDefaultReplication(path);
}
+ @Override
+ public boolean equals(Object o) {
+if (this == o) return true;
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command
CarbonDataQA1 commented on issue #3565: [CARBONDATA-3662]: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-572986951 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1583/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3566: [CARBONDATA-3492]: Added prepriming in the Index Server Documentation
CarbonDataQA1 commented on issue #3566: [CARBONDATA-3492]: Added prepriming in the Index Server Documentation URL: https://github.com/apache/carbondata/pull/3566#issuecomment-572975332 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1581/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365147091
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -135,23 +193,46 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
carbonTables.flatMap {
mainTable =>
try {
-makeRows(getTableCacheFromIndexServer(mainTable)(sparkSession),
mainTable)
+val row = makeRows(getTableCacheFromIndexServer(showExecutorCache,
+ mainTable)(sparkSession), mainTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
+}
+if (res(1) == 0 && res(2) == 0 && res(3) == 0) {
+ Seq()
+} else {
+ row
+}
} catch {
case ex: UnsupportedOperationException => Seq()
}
}
-} else { Seq() }
+} else {
+ Seq()
+}
val driverRows = if (cache != null) {
carbonTables.flatMap {
carbonTable =>
try {
-makeRows(getTableCacheFromDriver(sparkSession, carbonTable),
carbonTable)
+val row = makeRows(getTableCacheFromDriver(sparkSession,
carbonTable), carbonTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
Review comment:
No need to convert to List. same can be done for Seq also
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365146807
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -135,23 +193,46 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
carbonTables.flatMap {
mainTable =>
try {
-makeRows(getTableCacheFromIndexServer(mainTable)(sparkSession),
mainTable)
+val row = makeRows(getTableCacheFromIndexServer(showExecutorCache,
+ mainTable)(sparkSession), mainTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
+}
+if (res(1) == 0 && res(2) == 0 && res(3) == 0) {
+ Seq()
+} else {
+ row
+}
} catch {
case ex: UnsupportedOperationException => Seq()
}
}
-} else { Seq() }
+} else {
+ Seq()
+}
val driverRows = if (cache != null) {
carbonTables.flatMap {
carbonTable =>
try {
-makeRows(getTableCacheFromDriver(sparkSession, carbonTable),
carbonTable)
+val row = makeRows(getTableCacheFromDriver(sparkSession,
carbonTable), carbonTable)
+var res: List[Any] = null
+for (i <- row.toList) {
+ res = i.toSeq.toList
+}
+if (res(1) == 0 && res(2) == 0 && res(3) == 0) {
Review comment:
can be done in makeRows method
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365146606
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -171,39 +252,33 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
.toList)
if (driverRows.nonEmpty) {
(Seq(
- Row("ALL", "ALL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER"),
- Row(currentDatabase,
-"ALL",
-driverdbIndexSize,
-driverdbDatamapSize,
-driverdbDictSize,
-"DRIVER")
+ Row("TOTAL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER")
) ++ driverRows).collect {
- case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
-Row(row(0), row(1), bytesToDisplaySize(row.getLong(2)),
- bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "DRIVER")
+ case row if row.getLong(1) != 0L || row.getLong(2) != 0L ||
row.getLong(3) != 0L =>
+Row(row(0), bytesToDisplaySize(row.getLong(1)),
+ bytesToDisplaySize(row.getLong(2)),
bytesToDisplaySize(row.getLong(3)), "DRIVER")
}
} else {
-makeEmptyCacheRows(currentDatabase)
+makeEmptyCacheRows(currentDatabase, "DRIVER")
}
} else {
- makeEmptyCacheRows(currentDatabase)
+ makeEmptyCacheRows(currentDatabase, "DRIVER")
}
-// val (serverIndexSize, serverDataMapSize) =
getAllIndexServerCacheSize
-val indexDisplayRows = if (indexServerRows.nonEmpty) {
- (Seq(
-Row("ALL", "ALL", indexAllIndexSize, indexAllDatamapSize,
indexAllDictSize, "INDEX SERVER"),
-Row(currentDatabase,
- "ALL",
- indexdbIndexSize,
- indexdbDatamapSize,
- driverdbDictSize,
- "INDEX SERVER")
- ) ++ indexServerRows).collect {
-case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
- Row(row.get(0), row.get(1), bytesToDisplaySize(row.getLong(2)),
-bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "INDEX SERVER")
+val indexDisplayRows = if (isDistributedPruningEnabled) {
+ if (indexServerRows.nonEmpty) {
+(Seq(
+ Row("TOTAL", indexAllIndexSize, indexAllDatamapSize,
indexAllDictSize, "INDEX SERVER")
+) ++ indexServerRows).collect {
+ case row if row.getLong(1) != 0L || row.getLong(2) != 0L ||
row.getLong(3) != 0L =>
+Row(row.get(0),
+ bytesToDisplaySize(row.getLong(1)),
+ bytesToDisplaySize(row.getLong(2)),
+ bytesToDisplaySize(row.getLong(3)),
+ "INDEX SERVER")
+}
+ } else {
+makeEmptyCacheRows(currentDatabase, "INDEXSERVER")
Review comment:
If there is nothing in the cache then no need to display anything.
We can remove this method itself and simplyfy the conditions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] asfgit closed pull request #3573: [CARBONDATA-3661] Fix target file size check fail when upload local file to carbon store
asfgit closed pull request #3573: [CARBONDATA-3661] Fix target file size check fail when upload local file to carbon store URL: https://github.com/apache/carbondata/pull/3573 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365144756
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -171,39 +252,33 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
.toList)
if (driverRows.nonEmpty) {
(Seq(
- Row("ALL", "ALL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER"),
- Row(currentDatabase,
-"ALL",
-driverdbIndexSize,
-driverdbDatamapSize,
-driverdbDictSize,
-"DRIVER")
+ Row("TOTAL", driverIndexSize, driverDatamapSize, allDictSize,
"DRIVER")
) ++ driverRows).collect {
- case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
-Row(row(0), row(1), bytesToDisplaySize(row.getLong(2)),
- bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "DRIVER")
+ case row if row.getLong(1) != 0L || row.getLong(2) != 0L ||
row.getLong(3) != 0L =>
+Row(row(0), bytesToDisplaySize(row.getLong(1)),
+ bytesToDisplaySize(row.getLong(2)),
bytesToDisplaySize(row.getLong(3)), "DRIVER")
}
} else {
-makeEmptyCacheRows(currentDatabase)
+makeEmptyCacheRows(currentDatabase, "DRIVER")
}
} else {
- makeEmptyCacheRows(currentDatabase)
+ makeEmptyCacheRows(currentDatabase, "DRIVER")
}
-// val (serverIndexSize, serverDataMapSize) =
getAllIndexServerCacheSize
-val indexDisplayRows = if (indexServerRows.nonEmpty) {
- (Seq(
-Row("ALL", "ALL", indexAllIndexSize, indexAllDatamapSize,
indexAllDictSize, "INDEX SERVER"),
-Row(currentDatabase,
- "ALL",
- indexdbIndexSize,
- indexdbDatamapSize,
- driverdbDictSize,
- "INDEX SERVER")
- ) ++ indexServerRows).collect {
-case row if row.getLong(2) != 0L || row.getLong(3) != 0L ||
row.getLong(4) != 0L =>
- Row(row.get(0), row.get(1), bytesToDisplaySize(row.getLong(2)),
-bytesToDisplaySize(row.getLong(3)),
bytesToDisplaySize(row.getLong(4)), "INDEX SERVER")
+val indexDisplayRows = if (isDistributedPruningEnabled) {
+ if (indexServerRows.nonEmpty) {
Review comment:
revert unnecessary changes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365141311
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -127,6 +183,8 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
carbonTables += carbonTable
}
} catch {
+ case ex: AnalysisException =>
+LOGGER.info("Unable to access Carbon table object for table" +
tableIdent.table)
Review comment:
make this log as debug
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365139875
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -110,13 +137,42 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
}
}
+ def getAllExecutorCache(sparkSession: SparkSession): Seq[Row] = {
+val isDistributedPruningEnabled = CarbonProperties.getInstance()
+ .isDistributedPruningEnabled("", "")
+if (!isDistributedPruningEnabled) {
+ getAllTablesCache(sparkSession)
+}
+else {
+ // get all the executor details from the index server
+ try {
+val executorCacheValue = executeJobToGetCache(showExecutorCache,
List())
+val result = executorCacheValue.flatMap {
+ iterator =>
+Seq(Row(iterator._1, bytesToDisplaySize(iterator._3)))
+}
+result
+ }
+ catch {
+case ex: Exception =>
+ LOGGER.error("Error while getting cache from the Index Server", ex)
+ Seq()
+ }
+}
+ }
+
def getAllTablesCache(sparkSession: SparkSession): Seq[Row] = {
val currentDatabase = sparkSession.sessionState.catalog.getCurrentDatabase
val cache = CacheProvider.getInstance().getCarbonCache
val isDistributedPruningEnabled = CarbonProperties.getInstance()
.isDistributedPruningEnabled("", "")
-if (cache == null && !isDistributedPruningEnabled) {
- return makeEmptyCacheRows(currentDatabase)
+if (!isDistributedPruningEnabled) {
+ if (cache == null) {
+return makeEmptyCacheRows(currentDatabase, "DRIVER")
+ }
+ if (cache.getCurrentSize == 0) {
Review comment:
make as if (cache == null || cache.getCurrentSize == 0) {
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365138382
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -110,13 +137,42 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
}
}
+ def getAllExecutorCache(sparkSession: SparkSession): Seq[Row] = {
+val isDistributedPruningEnabled = CarbonProperties.getInstance()
+ .isDistributedPruningEnabled("", "")
+if (!isDistributedPruningEnabled) {
+ getAllTablesCache(sparkSession)
+}
+else {
Review comment:
fix the indentation for the whole PR
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365136701
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -41,25 +42,42 @@ import org.apache.carbondata.spark.util.CarbonScalaUtil
import org.apache.carbondata.spark.util.CommonUtil.bytesToDisplaySize
-case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
+case class CarbonShowCacheCommand(showExecutorCache: Boolean,
+tableIdentifier: Option[TableIdentifier],
internalCall: Boolean = false)
extends MetadataCommand {
private lazy val cacheResult: Seq[(String, Int, Long, String)] = {
-executeJobToGetCache(List())
+executeJobToGetCache(showExecutorCache, List())
}
private val LOGGER =
LogServiceFactory.getLogService(classOf[CarbonShowCacheCommand].getName)
override def output: Seq[AttributeReference] = {
if (tableIdentifier.isEmpty) {
- Seq(
-AttributeReference("Database", StringType, nullable = false)(),
-AttributeReference("Table", StringType, nullable = false)(),
-AttributeReference("Index size", StringType, nullable = false)(),
-AttributeReference("Datamap size", StringType, nullable = false)(),
-AttributeReference("Dictionary size", StringType, nullable = false)(),
-AttributeReference("Cache Location", StringType, nullable = false)())
+ val isDistributedPruningEnabled = CarbonProperties.getInstance()
+.isDistributedPruningEnabled("", "")
+ if (showExecutorCache == false) {
Review comment:
1. simplyfy the if else block
2. Block in case of indexserver=false
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] CarbonDataQA1 commented on issue #3466: [CARBONDATA-3586] [CARBONDATA-3587] [CARBONDATA-3595]:Adding valid segments into segments to be refreshed map before inserting segments to
CarbonDataQA1 commented on issue #3466: [CARBONDATA-3586] [CARBONDATA-3587] [CARBONDATA-3595]:Adding valid segments into segments to be refreshed map before inserting segments to index server URL: https://github.com/apache/carbondata/pull/3466#issuecomment-572941326 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/1579/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365133227
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -41,25 +42,42 @@ import org.apache.carbondata.spark.util.CarbonScalaUtil
import org.apache.carbondata.spark.util.CommonUtil.bytesToDisplaySize
-case class CarbonShowCacheCommand(tableIdentifier: Option[TableIdentifier],
+case class CarbonShowCacheCommand(showExecutorCache: Boolean,
+tableIdentifier: Option[TableIdentifier],
internalCall: Boolean = false)
extends MetadataCommand {
private lazy val cacheResult: Seq[(String, Int, Long, String)] = {
-executeJobToGetCache(List())
+executeJobToGetCache(showExecutorCache, List())
}
private val LOGGER =
LogServiceFactory.getLogService(classOf[CarbonShowCacheCommand].getName)
override def output: Seq[AttributeReference] = {
if (tableIdentifier.isEmpty) {
- Seq(
-AttributeReference("Database", StringType, nullable = false)(),
-AttributeReference("Table", StringType, nullable = false)(),
-AttributeReference("Index size", StringType, nullable = false)(),
-AttributeReference("Datamap size", StringType, nullable = false)(),
-AttributeReference("Dictionary size", StringType, nullable = false)(),
-AttributeReference("Cache Location", StringType, nullable = false)())
+ val isDistributedPruningEnabled = CarbonProperties.getInstance()
+.isDistributedPruningEnabled("", "")
+ if (showExecutorCache == false) {
+Seq(
+ AttributeReference("Database and Table", StringType, nullable =
false)(),
+ AttributeReference("Index size", StringType, nullable = false)(),
+ AttributeReference("Datamap size", StringType, nullable = false)(),
+ AttributeReference("Dictionary size", StringType, nullable =
false)(),
+ AttributeReference("Cache Location", StringType, nullable = false)())
+ } else {
+if (!isDistributedPruningEnabled) {
+ Seq(
+AttributeReference("Database and Table", StringType, nullable =
false)(),
Review comment:
Change "Database and Table" to "Identifier"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365130999
##
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/cache/CarbonShowCacheCommand.scala
##
@@ -74,7 +92,15 @@ case class CarbonShowCacheCommand(tableIdentifier:
Option[TableIdentifier],
/**
* Assemble result for database
*/
- getAllTablesCache(sparkSession)
+ if (showExecutorCache == false) {
Review comment:
"==false " not required...Change in all other places also
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365130483
##
File path:
integration/spark2/src/main/scala/org/apache/carbondata/indexserver/DistributedShowCacheRDD.scala
##
@@ -71,10 +75,21 @@ class DistributedShowCacheRDD(@transient private val ss:
SparkSession, tableUniq
.getTableUniqueName
} else {
dataMap.getDataMapSchema.getRelationIdentifier.getDatabaseName +
"_" + dataMap
- .getDataMapSchema.getDataMapName
+.getDataMapSchema.getDataMapName
+}
+if (executorCache == true) {
Review comment:
"== true " is not required
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365130789
##
File path:
integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
##
@@ -205,14 +205,18 @@ object IndexServer extends ServerInterface {
}
}
- override def showCache(tableId: String = ""): Array[String] = doAs {
+ override def showCache(executorCache: Boolean, tableId: String = ""):
Array[String] = doAs {
val jobgroup: String = "Show Cache " + (tableId match {
- case "" => "for all tables"
+ case "" =>
+executorCache match {
Review comment:
replace with if else
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]: Changes to show metacache command
kunal642 commented on a change in pull request #3565: [CARBONDATA-3662]:
Changes to show metacache command
URL: https://github.com/apache/carbondata/pull/3565#discussion_r365130708
##
File path:
integration/spark2/src/main/scala/org/apache/carbondata/indexserver/IndexServer.scala
##
@@ -57,7 +57,7 @@ trait ServerInterface {
/**
* Get the cache size for the specified tables.
*/
- def showCache(tableIds: String) : Array[String]
+ def showCache(executorCache: Boolean, tableIds: String) : Array[String]
Review comment:
Add to the end
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on a change in pull request #3566: [CARBONDATA-3492]: Added prepriming in the Index Server Documentation
vikramahuja1001 commented on a change in pull request #3566: [CARBONDATA-3492]: Added prepriming in the Index Server Documentation URL: https://github.com/apache/carbondata/pull/3566#discussion_r365127281 ## File path: docs/index-server.md ## @@ -119,6 +119,20 @@ be written to file. The user can set the location for these file by using 'carbon.indexserver.temp.path'. By default Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (CARBONDATA-3662) Changes to Show metacache command
[ https://issues.apache.org/jira/browse/CARBONDATA-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikram Ahuja updated CARBONDATA-3662: - Description: The following changes are proposed to the show metacache command: # There is no command to show the cache occupied by each of the executors(only applicable for index server). This command will not show per table, only per node Example: Driver - X bytes Executor1 - Y bytes Executor2 - Z bytes To change _show metacache_ command to _show executor metacache_, where the "executor" keyword is optional. # Show metacache command does not show the complete cache used used by IndexServer side and Driver side. It just shows the total of both of them. Refer: [http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Changes-to-SHOW-METACACHE-command-td87433.html] was: The following changes are proposed to the show metacache command: # There is no command to show the cache occupied by each of the executors(only applicable for index server). This command will not show per table, only per node Example: Driver - X bytes Executor1 - Y bytes Executor2 - Z bytes To change _show metacache_ command to _show executor metacache_, where the "executor" keyword is optional. # Show metacache command does not show the complete cache used used by IndexServer side and Driver side. It just shows the total of both of them. > Changes to Show metacache command > - > > Key: CARBONDATA-3662 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3662 > Project: CarbonData > Issue Type: Bug > Components: spark-integration >Affects Versions: 2.0.0 >Reporter: Vikram Ahuja >Priority: Minor > > The following changes are proposed to the show metacache command: > # There is no command to show the cache occupied by each of the > executors(only applicable for index server). > This command will not show per table, only per node > Example: > Driver - X bytes > Executor1 - Y bytes > Executor2 - Z bytes > To change _show metacache_ command to _show executor metacache_, where the > "executor" keyword is optional. > # Show metacache command does not show the complete cache used used by > IndexServer side and Driver side. It just shows the total of both of them. > Refer: > [http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/DISCUSSION-Changes-to-SHOW-METACACHE-command-td87433.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3662) Changes to Show metacache command
Vikram Ahuja created CARBONDATA-3662: Summary: Changes to Show metacache command Key: CARBONDATA-3662 URL: https://issues.apache.org/jira/browse/CARBONDATA-3662 Project: CarbonData Issue Type: Bug Components: spark-integration Affects Versions: 2.0.0 Reporter: Vikram Ahuja The following changes are proposed to the show metacache command: # There is no command to show the cache occupied by each of the executors(only applicable for index server). This command will not show per table, only per node Example: Driver - X bytes Executor1 - Y bytes Executor2 - Z bytes To change _show metacache_ command to _show executor metacache_, where the "executor" keyword is optional. # Show metacache command does not show the complete cache used used by IndexServer side and Driver side. It just shows the total of both of them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [carbondata] chetandb commented on a change in pull request #3566: [CARBONDATA-3492]: Added prepriming in the Index Server Documentation
chetandb commented on a change in pull request #3566: [CARBONDATA-3492]: Added prepriming in the Index Server Documentation URL: https://github.com/apache/carbondata/pull/3566#discussion_r365120979 ## File path: docs/index-server.md ## @@ -119,6 +119,20 @@ be written to file. The user can set the location for these file by using 'carbon.indexserver.temp.path'. By default Review comment: "location for these file" to be changed to "location for these files" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on issue #3565: Changes to show metacache command
vikramahuja1001 commented on issue #3565: Changes to show metacache command URL: https://github.com/apache/carbondata/pull/3565#issuecomment-572920939 @kunal642 , @akashrn5 , please review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [carbondata] vikramahuja1001 commented on issue #3537: [CARBONDATA-3646] [CARBONDATA-3647]: Fix query failure with Index Server
vikramahuja1001 commented on issue #3537: [CARBONDATA-3646] [CARBONDATA-3647]: Fix query failure with Index Server URL: https://github.com/apache/carbondata/pull/3537#issuecomment-572920734 @kunal642 , @akashrn5 , please give LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
