[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[ https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595670#comment-15595670 ] ASF GitHub Bot commented on CARBONDATA-306: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/230 > block size info should be show in Desc Formatted and executor log > - > > Key: CARBONDATA-306 > URL: https://issues.apache.org/jira/browse/CARBONDATA-306 > Project: CarbonData > Issue Type: Improvement >Reporter: Jay >Assignee: Jay >Priority: Minor > > when run desc formatted command, the table block size should be show, as well > as in executor log when run load command -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[ https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595666#comment-15595666 ] ASF GitHub Bot commented on CARBONDATA-306: --- Github user asfgit closed the pull request at: https://github.com/apache/incubator-carbondata/pull/230 > block size info should be show in Desc Formatted and executor log > - > > Key: CARBONDATA-306 > URL: https://issues.apache.org/jira/browse/CARBONDATA-306 > Project: CarbonData > Issue Type: Improvement >Reporter: Jay >Assignee: Jay >Priority: Minor > > when run desc formatted command, the table block size should be show, as well > as in executor log when run load command -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575341#comment-15575341
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83421288
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/mdkeygen/MDKeyGenStep.java
---
@@ -314,7 +314,7 @@ private boolean setStepConfiguration() {
wrapperColumnSchema = CarbonUtil
.getColumnSchemaList(carbonTable.getDimensionByTableName(tableName),
carbonTable.getMeasureByTableName(tableName));
-blocksize = carbonTable.getBlocksize();
+blocksize = carbonTable.getBlocksizeInMB();
--- End diff --
should be `getBlockSizeInMB`
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574635#comment-15574635
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user Jay357089 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83376650
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala
---
@@ -1422,6 +1422,7 @@ private[sql] case class DescribeCommandFormatted(
results ++= Seq(("Table Name : ",
relation.tableMeta.carbonTableIdentifier.getTableName, ""))
results ++= Seq(("CARBON Store Path : ", relation.tableMeta.storePath,
""))
val carbonTable = relation.tableMeta.carbonTable
+results ++= Seq(("Table Block Size : ", carbonTable.getBlocksize + "
MB", ""))
--- End diff --
done. CI passed.
http://136.243.101.176:8080/job/ApacheCarbonManualPRBuilder/429/
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574337#comment-15574337
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83361557
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala
---
@@ -1422,6 +1422,7 @@ private[sql] case class DescribeCommandFormatted(
results ++= Seq(("Table Name : ",
relation.tableMeta.carbonTableIdentifier.getTableName, ""))
results ++= Seq(("CARBON Store Path : ", relation.tableMeta.storePath,
""))
val carbonTable = relation.tableMeta.carbonTable
+results ++= Seq(("Table Block Size : ", carbonTable.getBlocksize + "
MB", ""))
--- End diff --
If so, can you change the corresponding variable name and function name to
indicate it is bytes in MB, like `getBlockSizeInMB` and add comment to
`CarbonCommonConstants.TABLE_BLOCKSIZE`
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574086#comment-15574086
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user Jay357089 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83354112
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,15 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+long setBlockSizeInMb = blockSize /
CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR /
+CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR;
+// actual file size may be less than 1KB or 1MB, need to classify.
+String readableFileSize = ByteUtil.convertByteToReadable(fileSize);
+long maxSizeInMb = maxSize /
CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR /
+CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR;
+LOGGER.info("The configured block size is " + setBlockSizeInMb + " MB,
" +
--- End diff --
done. CI passed.
http://136.243.101.176:8080/job/ApacheCarbonManualPRBuilder/427/
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15574011#comment-15574011
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user Jay357089 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83352166
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala
---
@@ -1422,6 +1422,7 @@ private[sql] case class DescribeCommandFormatted(
results ++= Seq(("Table Name : ",
relation.tableMeta.carbonTableIdentifier.getTableName, ""))
results ++= Seq(("CARBON Store Path : ", relation.tableMeta.storePath,
""))
val carbonTable = relation.tableMeta.carbonTable
+results ++= Seq(("Table Block Size : ", carbonTable.getBlocksize + "
MB", ""))
--- End diff --
in carbonTable, block size is set in MB, it's already readable, so i don't
think it need to format
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573920#comment-15573920
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83349683
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,15 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+long setBlockSizeInMb = blockSize /
CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR /
+CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR;
+// actual file size may be less than 1KB or 1MB, need to classify.
+String readableFileSize = ByteUtil.convertByteToReadable(fileSize);
+long maxSizeInMb = maxSize /
CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR /
+CarbonCommonConstants.BYTE_TO_KB_CONVERSION_FACTOR;
+LOGGER.info("The configured block size is " + setBlockSizeInMb + " MB,
" +
--- End diff --
Use the new function here to format readable
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15573919#comment-15573919
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83349721
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala
---
@@ -1422,6 +1422,7 @@ private[sql] case class DescribeCommandFormatted(
results ++= Seq(("Table Name : ",
relation.tableMeta.carbonTableIdentifier.getTableName, ""))
results ++= Seq(("CARBON Store Path : ", relation.tableMeta.storePath,
""))
val carbonTable = relation.tableMeta.carbonTable
+results ++= Seq(("Table Block Size : ", carbonTable.getBlocksize + "
MB", ""))
--- End diff --
Use the new function here to format readable
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571826#comment-15571826
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user Jay357089 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83208043
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+LOGGER.info("The configured block size is " + blockSize + " byte, " +
--- End diff --
@jackylk @Zhangshunyu done.
CI passed.
http://136.243.101.176:8080/job/ApacheCarbonManualPRBuilder/424/
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570706#comment-15570706
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83139950
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+LOGGER.info("The configured block size is " + blockSize + " byte, " +
--- End diff --
@Jay357089 I think this is a good idea to extract ConvertByteToReadable as
a method, since it can be used in many logs, especially for analyzing
performance.
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570699#comment-15570699
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user Jay357089 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83139600
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+LOGGER.info("The configured block size is " + blockSize + " byte, " +
--- End diff --
@jackylk Maybe i should extract if .. else part to a method called
ConvertByteToReadable, what's your opinion?
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569003#comment-15569003
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83028164
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+LOGGER.info("The configured block size is " + blockSize + " byte, " +
--- End diff --
Suggest to have the `blockSize` convert to a proper number before logging
it, otherwise it is hard to check this value by human
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568998#comment-15568998
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83027603
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+LOGGER.info("The configured block size is " + blockSize + " byte, " +
--- End diff --
@jackylk set in mb，but here already converted to byte.
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CARBONDATA-306) block size info should be show in Desc Formatted and executor log
[
https://issues.apache.org/jira/browse/CARBONDATA-306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568922#comment-15568922
]
ASF GitHub Bot commented on CARBONDATA-306:
---
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83021340
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private static long getMaxOfBlockAndFileSize(long
blockSize, long fileSize) {
if (remainder > 0) {
maxSize = maxSize + HDFS_CHECKSUM_LENGTH - remainder;
}
+LOGGER.info("The configured block size is " + blockSize + " byte, " +
--- End diff --
Is `blockSize` in bytes or MB?
> block size info should be show in Desc Formatted and executor log
> -
>
> Key: CARBONDATA-306
> URL: https://issues.apache.org/jira/browse/CARBONDATA-306
> Project: CarbonData
> Issue Type: Improvement
>Reporter: Jay
>Priority: Minor
>
> when run desc formatted command, the table block size should be show, as well
> as in executor log when run load command
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
