[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2875 ---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230991320
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -1744,89 +2021,101 @@ private CarbonCommonConstants() {
* BYTE_ENCODING
*/
public static final String BYTE_ENCODING = "ISO-8859-1";
+
--- End diff --
support configure carbon.push.rowfilters.for.vector in set command, dynamic
configurable in this PR
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629905
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -975,16 +1140,39 @@ private CarbonCommonConstants() {
* make it less and not the index writing speed.
*/
public static final String CARBON_LUCENE_COMPRESSION_MODE_DEFAULT =
"speed";
+
/**
* The node loads the smallest amount of data
*/
@CarbonProperty
public static final String CARBON_LOAD_MIN_SIZE_INMB =
"load_min_size_inmb";
+
+ /**
+ * the node minimum load data default value
+ */
public static final String CARBON_LOAD_MIN_SIZE_INMB_DEFAULT = "0";
+
+ @CarbonProperty
+ public static final String IN_MEMORY_FOR_SORT_DATA_IN_MB =
"sort.inmemory.size.inmb";
+
+ public static final String IN_MEMORY_FOR_SORT_DATA_IN_MB_DEFAULT =
"1024";
+
+ /**
+ * carbon sort size
+ */
+ @Deprecated
--- End diff --
ok
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629760
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -117,77 +126,106 @@ private CarbonCommonConstants() {
* CARBON_TIMESTAMP
*/
public static final String CARBON_TIMESTAMP_MILLIS = "dd-MM-
HH:mm:ss:SSS";
+
/**
- * compressor for writing/reading carbondata file
+ * Property for specifying the format of DATE data type column.
+ * e.g. /MM/dd , or using default value
*/
+ @CarbonProperty
+ public static final String CARBON_DATE_FORMAT = "carbon.date.format";
+
+ /**
+ * default value
+ */
+ public static final String CARBON_DATE_DEFAULT_FORMAT = "-MM-dd";
+
+ /**
+ * compressor for writing/reading CarbonData file
+ */
+ @CarbonProperty
public static final String COMPRESSOR = "carbon.column.compressor";
/**
* default compressor is snappy
*/
public static final String DEFAULT_COMPRESSOR = "snappy";
+
/**
- * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper will be
used to handle locking
+ * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper
+ * will be used to handle locking
* mechanism of carbon
*/
@CarbonProperty
public static final String LOCK_TYPE = "carbon.lock.type";
+ /**
+ * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
+ */
+ public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+
/**
* Specifies the path where the lock files have to be created.
* By default, lock files are created in table path.
*/
@CarbonProperty
public static final String LOCK_PATH = "carbon.lock.path";
- /**
- * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
- */
- public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+ public static final String LOCK_PATH_DEFAULT = "";
/**
* ZOOKEEPER_LOCATION this is the location in zookeeper file system
where locks are created.
* mechanism of carbon
*/
public static final String ZOOKEEPER_LOCATION = "/CarbonLocks";
+
/**
- * xxhash algorithm property for hashmap
+ * xxhash algorithm property for hashMap
*/
+ @CarbonProperty
public static final String ENABLE_XXHASH = "carbon.enableXXHash";
/**
- * xxhash algorithm property for hashmap. Default value false
+ * xxhash algorithm property for hashMap Default value false
*/
public static final String ENABLE_XXHASH_DEFAULT = "true";
+
/**
* System property to enable or disable local dictionary generation
*/
+ @CarbonProperty
public static final String LOCAL_DICTIONARY_SYSTEM_ENABLE =
"carbon.local.dictionary.enable";
/**
* System property to enable or disable decoder based local dictionary
fallback
*/
+ @CarbonProperty
public static final String LOCAL_DICTIONARY_DECODER_BASED_FALLBACK =
"carbon.local.dictionary.decoder.fallback";
/**
* System property to enable or disable decoder based local dictionary
fallback default value
*/
public static final String
LOCAL_DICTIONARY_DECODER_BASED_FALLBACK_DEFAULT = "true";
+
/**
* zookeeper url key
*/
+ @CarbonProperty
public static final String ZOOKEEPER_URL = "spark.deploy.zookeeper.url";
+
/**
* carbon data file version property
*/
+ @CarbonProperty
public static final String CARBON_DATA_FILE_VERSION =
"carbon.data.file.version";
+
@CarbonProperty
public static final String SPARK_SCHEMA_STRING_LENGTH_THRESHOLD =
"spark.sql.sources.schemaStringLengthThreshold";
public static final int SPARK_SCHEMA_STRING_LENGTH_THRESHOLD_DEFAULT =
4000;
+ //TODO: add annotation and whether change to dynamic
--- End diff --
ok, done
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629803
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -897,7 +1041,8 @@ private CarbonCommonConstants() {
/**
* property for configuring parallelism per segment when doing an
update. Increase this
* value will avoid data screw problem for a large segment.
- * Refer to CARBONDATA-1373 for more details.
+ * Refer to CarbonData-1373 for more details.
+ * TODO: dynamic
--- End diff --
removed, this property should can be dynamic configurable
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629792
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -435,53 +520,77 @@ private CarbonCommonConstants() {
public static final int SORT_INTERMEDIATE_FILES_LIMIT_MIN = 2;
public static final int SORT_INTERMEDIATE_FILES_LIMIT_MAX = 50;
+
/**
* SORT_FILE_WRITE_BUFFER_SIZE
*/
@CarbonProperty
public static final String CARBON_SORT_FILE_WRITE_BUFFER_SIZE =
"carbon.sort.file.write.buffer.size";
- public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MIN = 10240;
- public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MAX =
10485760;
+
/**
* SORT_FILE_WRITE_BUFFER_SIZE_DEFAULT_VALUE
*/
public static final String
CARBON_SORT_FILE_WRITE_BUFFER_SIZE_DEFAULT_VALUE = "16384";
+
+ public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MIN = 10240;
+
+ public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MAX =
10485760;
+
/**
* CSV_READ_BUFFER_SIZE
*/
@CarbonProperty
public static final String CSV_READ_BUFFER_SIZE =
"carbon.csv.read.buffersize.byte";
+
/**
* CSV_READ_BUFFER_SIZE
+ * default value is 1mb
*/
- public static final String CSV_READ_BUFFER_SIZE_DEFAULT = "1048576"; //
1mb
+ public static final String CSV_READ_BUFFER_SIZE_DEFAULT = "1048576";
+
/**
- * min value for csv read buffer size
+ * min value for csv read buffer size, 10 kb
*/
- public static final int CSV_READ_BUFFER_SIZE_MIN = 10240; //10 kb
+ public static final int CSV_READ_BUFFER_SIZE_MIN = 10240;
+
/**
- * max value for csv read buffer size
+ * max value for csv read buffer size, 10 mb
*/
- public static final int CSV_READ_BUFFER_SIZE_MAX = 10485760; // 10 mb
+ public static final int CSV_READ_BUFFER_SIZE_MAX = 10485760;
+
/**
* CARBON_MERGE_SORT_READER_THREAD
*/
@CarbonProperty
public static final String CARBON_MERGE_SORT_READER_THREAD =
"carbon.merge.sort.reader.thread";
+
/**
- * CARBON_MERGE_SORT_READER_THREAD_DEFAULTVALUE
+ * CARBON_MERGE_SORT_READER_THREAD DEFAULT value
*/
public static final String CARBON_MERGE_SORT_READER_THREAD_DEFAULTVALUE
= "3";
+
/**
* TIME_STAT_UTIL_TYPE
+ * TODO:dynamic
*/
@CarbonProperty
public static final String ENABLE_DATA_LOADING_STATISTICS =
"enable.data.loading.statistics";
+
/**
* TIME_STAT_UTIL_TYPE_DEFAULT
*/
+
public static final String ENABLE_DATA_LOADING_STATISTICS_DEFAULT =
"false";
+
+ /**
+ * NUMBER_OF_TRIES_FOR_LOAD_METADATA_LOCK
+ * TODO: dynamic
--- End diff --
removed, this property should can be dynamic configurable
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629800
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -594,6 +713,7 @@ private CarbonCommonConstants() {
/**
* This property will determine the loads of how many days can be
compacted.
+ * TODO: dynamic
--- End diff --
removed, this property should can be dynamic configurable
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629669
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -117,77 +126,106 @@ private CarbonCommonConstants() {
* CARBON_TIMESTAMP
*/
public static final String CARBON_TIMESTAMP_MILLIS = "dd-MM-
HH:mm:ss:SSS";
+
/**
- * compressor for writing/reading carbondata file
+ * Property for specifying the format of DATE data type column.
+ * e.g. /MM/dd , or using default value
*/
+ @CarbonProperty
+ public static final String CARBON_DATE_FORMAT = "carbon.date.format";
+
+ /**
+ * default value
+ */
+ public static final String CARBON_DATE_DEFAULT_FORMAT = "-MM-dd";
+
+ /**
+ * compressor for writing/reading CarbonData file
+ */
+ @CarbonProperty
public static final String COMPRESSOR = "carbon.column.compressor";
/**
* default compressor is snappy
*/
public static final String DEFAULT_COMPRESSOR = "snappy";
+
/**
- * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper will be
used to handle locking
+ * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper
+ * will be used to handle locking
* mechanism of carbon
*/
@CarbonProperty
public static final String LOCK_TYPE = "carbon.lock.type";
+ /**
+ * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
+ */
+ public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+
/**
* Specifies the path where the lock files have to be created.
* By default, lock files are created in table path.
*/
@CarbonProperty
public static final String LOCK_PATH = "carbon.lock.path";
- /**
- * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
- */
- public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+ public static final String LOCK_PATH_DEFAULT = "";
/**
* ZOOKEEPER_LOCATION this is the location in zookeeper file system
where locks are created.
* mechanism of carbon
*/
public static final String ZOOKEEPER_LOCATION = "/CarbonLocks";
+
/**
- * xxhash algorithm property for hashmap
+ * xxhash algorithm property for hashMap
*/
+ @CarbonProperty
public static final String ENABLE_XXHASH = "carbon.enableXXHash";
/**
- * xxhash algorithm property for hashmap. Default value false
+ * xxhash algorithm property for hashMap Default value false
--- End diff --
ok, keep
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629658
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -117,77 +126,106 @@ private CarbonCommonConstants() {
* CARBON_TIMESTAMP
*/
public static final String CARBON_TIMESTAMP_MILLIS = "dd-MM-
HH:mm:ss:SSS";
+
/**
- * compressor for writing/reading carbondata file
+ * Property for specifying the format of DATE data type column.
+ * e.g. /MM/dd , or using default value
*/
+ @CarbonProperty
+ public static final String CARBON_DATE_FORMAT = "carbon.date.format";
+
+ /**
+ * default value
+ */
+ public static final String CARBON_DATE_DEFAULT_FORMAT = "-MM-dd";
+
+ /**
+ * compressor for writing/reading CarbonData file
+ */
+ @CarbonProperty
public static final String COMPRESSOR = "carbon.column.compressor";
/**
* default compressor is snappy
*/
public static final String DEFAULT_COMPRESSOR = "snappy";
+
/**
- * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper will be
used to handle locking
+ * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper
+ * will be used to handle locking
* mechanism of carbon
*/
@CarbonProperty
public static final String LOCK_TYPE = "carbon.lock.type";
+ /**
+ * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
+ */
+ public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+
/**
* Specifies the path where the lock files have to be created.
* By default, lock files are created in table path.
*/
@CarbonProperty
public static final String LOCK_PATH = "carbon.lock.path";
- /**
- * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
- */
- public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+ public static final String LOCK_PATH_DEFAULT = "";
/**
* ZOOKEEPER_LOCATION this is the location in zookeeper file system
where locks are created.
* mechanism of carbon
*/
public static final String ZOOKEEPER_LOCATION = "/CarbonLocks";
+
/**
- * xxhash algorithm property for hashmap
+ * xxhash algorithm property for hashMap
--- End diff --
ok,done. We can add it in next PR, add .usedBy("") for it.
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230629266
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/HDFSLeaseUtils.java ---
@@ -164,22 +157,25 @@ private static boolean recoverLeaseOnFile(String
filePath, Path path, Distribute
private static int getLeaseRecoveryRetryCount() {
String retryMaxAttempts = CarbonProperties.getInstance()
.getProperty(CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT,
-CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
+
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
int retryCount = 0;
try {
retryCount = Integer.parseInt(retryMaxAttempts);
- if (retryCount < CARBON_LEASE_RECOVERY_RETRY_COUNT_MIN
- || retryCount > CARBON_LEASE_RECOVERY_RETRY_COUNT_MAX) {
-retryCount =
Integer.parseInt(CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
+ if (retryCount <
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_MIN
+ || retryCount >
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_MAX) {
+retryCount = Integer.parseInt(
+
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
LOGGER.warn(
"value configured for " +
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT
--- End diff --
OKï¼done
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230626185
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -722,17 +723,19 @@ private void loadProperties() {
}
} catch (FileNotFoundException e) {
LOGGER.error(
- "The file: " + CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH
+ " does not exist");
+ "The file: " +
CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH_DEFAULT
--- End diff --
OKï¼down
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230626228
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -722,17 +723,19 @@ private void loadProperties() {
}
} catch (FileNotFoundException e) {
LOGGER.error(
- "The file: " + CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH
+ " does not exist");
+ "The file: " +
CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH_DEFAULT
+ + " does not exist");
} catch (IOException e) {
LOGGER.error(
- "Error while reading the file: " +
CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH);
+ "Error while reading the file: "
+ + CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH_DEFAULT);
--- End diff --
OKï¼done
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230618777
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -1744,89 +2021,101 @@ private CarbonCommonConstants() {
* BYTE_ENCODING
*/
public static final String BYTE_ENCODING = "ISO-8859-1";
+
--- End diff --
carbon.push.rowfilters.for.vector is not in correct place and it should be
dynamic=true
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230618624
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -975,16 +1140,39 @@ private CarbonCommonConstants() {
* make it less and not the index writing speed.
*/
public static final String CARBON_LUCENE_COMPRESSION_MODE_DEFAULT =
"speed";
+
/**
* The node loads the smallest amount of data
*/
@CarbonProperty
public static final String CARBON_LOAD_MIN_SIZE_INMB =
"load_min_size_inmb";
+
+ /**
+ * the node minimum load data default value
+ */
public static final String CARBON_LOAD_MIN_SIZE_INMB_DEFAULT = "0";
+
+ @CarbonProperty
+ public static final String IN_MEMORY_FOR_SORT_DATA_IN_MB =
"sort.inmemory.size.inmb";
+
+ public static final String IN_MEMORY_FOR_SORT_DATA_IN_MB_DEFAULT =
"1024";
+
+ /**
+ * carbon sort size
+ */
+ @Deprecated
--- End diff --
Seems it is still used in SortDataRows, so it is not deprecated
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230618503
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -897,7 +1041,8 @@ private CarbonCommonConstants() {
/**
* property for configuring parallelism per segment when doing an
update. Increase this
* value will avoid data screw problem for a large segment.
- * Refer to CARBONDATA-1373 for more details.
+ * Refer to CarbonData-1373 for more details.
+ * TODO: dynamic
--- End diff --
?
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230618427
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -594,6 +713,7 @@ private CarbonCommonConstants() {
/**
* This property will determine the loads of how many days can be
compacted.
+ * TODO: dynamic
--- End diff --
?
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230618398
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -435,53 +520,77 @@ private CarbonCommonConstants() {
public static final int SORT_INTERMEDIATE_FILES_LIMIT_MIN = 2;
public static final int SORT_INTERMEDIATE_FILES_LIMIT_MAX = 50;
+
/**
* SORT_FILE_WRITE_BUFFER_SIZE
*/
@CarbonProperty
public static final String CARBON_SORT_FILE_WRITE_BUFFER_SIZE =
"carbon.sort.file.write.buffer.size";
- public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MIN = 10240;
- public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MAX =
10485760;
+
/**
* SORT_FILE_WRITE_BUFFER_SIZE_DEFAULT_VALUE
*/
public static final String
CARBON_SORT_FILE_WRITE_BUFFER_SIZE_DEFAULT_VALUE = "16384";
+
+ public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MIN = 10240;
+
+ public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MAX =
10485760;
+
/**
* CSV_READ_BUFFER_SIZE
*/
@CarbonProperty
public static final String CSV_READ_BUFFER_SIZE =
"carbon.csv.read.buffersize.byte";
+
/**
* CSV_READ_BUFFER_SIZE
+ * default value is 1mb
*/
- public static final String CSV_READ_BUFFER_SIZE_DEFAULT = "1048576"; //
1mb
+ public static final String CSV_READ_BUFFER_SIZE_DEFAULT = "1048576";
+
/**
- * min value for csv read buffer size
+ * min value for csv read buffer size, 10 kb
*/
- public static final int CSV_READ_BUFFER_SIZE_MIN = 10240; //10 kb
+ public static final int CSV_READ_BUFFER_SIZE_MIN = 10240;
+
/**
- * max value for csv read buffer size
+ * max value for csv read buffer size, 10 mb
*/
- public static final int CSV_READ_BUFFER_SIZE_MAX = 10485760; // 10 mb
+ public static final int CSV_READ_BUFFER_SIZE_MAX = 10485760;
+
/**
* CARBON_MERGE_SORT_READER_THREAD
*/
@CarbonProperty
public static final String CARBON_MERGE_SORT_READER_THREAD =
"carbon.merge.sort.reader.thread";
+
/**
- * CARBON_MERGE_SORT_READER_THREAD_DEFAULTVALUE
+ * CARBON_MERGE_SORT_READER_THREAD DEFAULT value
*/
public static final String CARBON_MERGE_SORT_READER_THREAD_DEFAULTVALUE
= "3";
+
/**
* TIME_STAT_UTIL_TYPE
+ * TODO:dynamic
*/
@CarbonProperty
public static final String ENABLE_DATA_LOADING_STATISTICS =
"enable.data.loading.statistics";
+
/**
* TIME_STAT_UTIL_TYPE_DEFAULT
*/
+
public static final String ENABLE_DATA_LOADING_STATISTICS_DEFAULT =
"false";
+
+ /**
+ * NUMBER_OF_TRIES_FOR_LOAD_METADATA_LOCK
+ * TODO: dynamic
--- End diff --
?
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230618379
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -435,53 +520,77 @@ private CarbonCommonConstants() {
public static final int SORT_INTERMEDIATE_FILES_LIMIT_MIN = 2;
public static final int SORT_INTERMEDIATE_FILES_LIMIT_MAX = 50;
+
/**
* SORT_FILE_WRITE_BUFFER_SIZE
*/
@CarbonProperty
public static final String CARBON_SORT_FILE_WRITE_BUFFER_SIZE =
"carbon.sort.file.write.buffer.size";
- public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MIN = 10240;
- public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MAX =
10485760;
+
/**
* SORT_FILE_WRITE_BUFFER_SIZE_DEFAULT_VALUE
*/
public static final String
CARBON_SORT_FILE_WRITE_BUFFER_SIZE_DEFAULT_VALUE = "16384";
+
+ public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MIN = 10240;
+
+ public static final int CARBON_SORT_FILE_WRITE_BUFFER_SIZE_MAX =
10485760;
+
/**
* CSV_READ_BUFFER_SIZE
*/
@CarbonProperty
public static final String CSV_READ_BUFFER_SIZE =
"carbon.csv.read.buffersize.byte";
+
/**
* CSV_READ_BUFFER_SIZE
+ * default value is 1mb
*/
- public static final String CSV_READ_BUFFER_SIZE_DEFAULT = "1048576"; //
1mb
+ public static final String CSV_READ_BUFFER_SIZE_DEFAULT = "1048576";
+
/**
- * min value for csv read buffer size
+ * min value for csv read buffer size, 10 kb
*/
- public static final int CSV_READ_BUFFER_SIZE_MIN = 10240; //10 kb
+ public static final int CSV_READ_BUFFER_SIZE_MIN = 10240;
+
/**
- * max value for csv read buffer size
+ * max value for csv read buffer size, 10 mb
*/
- public static final int CSV_READ_BUFFER_SIZE_MAX = 10485760; // 10 mb
+ public static final int CSV_READ_BUFFER_SIZE_MAX = 10485760;
+
/**
* CARBON_MERGE_SORT_READER_THREAD
*/
@CarbonProperty
public static final String CARBON_MERGE_SORT_READER_THREAD =
"carbon.merge.sort.reader.thread";
+
/**
- * CARBON_MERGE_SORT_READER_THREAD_DEFAULTVALUE
+ * CARBON_MERGE_SORT_READER_THREAD DEFAULT value
*/
public static final String CARBON_MERGE_SORT_READER_THREAD_DEFAULTVALUE
= "3";
+
/**
* TIME_STAT_UTIL_TYPE
+ * TODO:dynamic
--- End diff --
?
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230617961
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -117,77 +126,106 @@ private CarbonCommonConstants() {
* CARBON_TIMESTAMP
*/
public static final String CARBON_TIMESTAMP_MILLIS = "dd-MM-
HH:mm:ss:SSS";
+
/**
- * compressor for writing/reading carbondata file
+ * Property for specifying the format of DATE data type column.
+ * e.g. /MM/dd , or using default value
*/
+ @CarbonProperty
+ public static final String CARBON_DATE_FORMAT = "carbon.date.format";
+
+ /**
+ * default value
+ */
+ public static final String CARBON_DATE_DEFAULT_FORMAT = "-MM-dd";
+
+ /**
+ * compressor for writing/reading CarbonData file
+ */
+ @CarbonProperty
public static final String COMPRESSOR = "carbon.column.compressor";
/**
* default compressor is snappy
*/
public static final String DEFAULT_COMPRESSOR = "snappy";
+
/**
- * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper will be
used to handle locking
+ * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper
+ * will be used to handle locking
* mechanism of carbon
*/
@CarbonProperty
public static final String LOCK_TYPE = "carbon.lock.type";
+ /**
+ * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
+ */
+ public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+
/**
* Specifies the path where the lock files have to be created.
* By default, lock files are created in table path.
*/
@CarbonProperty
public static final String LOCK_PATH = "carbon.lock.path";
- /**
- * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
- */
- public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+ public static final String LOCK_PATH_DEFAULT = "";
/**
* ZOOKEEPER_LOCATION this is the location in zookeeper file system
where locks are created.
* mechanism of carbon
*/
public static final String ZOOKEEPER_LOCATION = "/CarbonLocks";
+
/**
- * xxhash algorithm property for hashmap
+ * xxhash algorithm property for hashMap
*/
+ @CarbonProperty
public static final String ENABLE_XXHASH = "carbon.enableXXHash";
/**
- * xxhash algorithm property for hashmap. Default value false
+ * xxhash algorithm property for hashMap Default value false
*/
public static final String ENABLE_XXHASH_DEFAULT = "true";
+
/**
* System property to enable or disable local dictionary generation
*/
+ @CarbonProperty
public static final String LOCAL_DICTIONARY_SYSTEM_ENABLE =
"carbon.local.dictionary.enable";
/**
* System property to enable or disable decoder based local dictionary
fallback
*/
+ @CarbonProperty
public static final String LOCAL_DICTIONARY_DECODER_BASED_FALLBACK =
"carbon.local.dictionary.decoder.fallback";
/**
* System property to enable or disable decoder based local dictionary
fallback default value
*/
public static final String
LOCAL_DICTIONARY_DECODER_BASED_FALLBACK_DEFAULT = "true";
+
/**
* zookeeper url key
*/
+ @CarbonProperty
public static final String ZOOKEEPER_URL = "spark.deploy.zookeeper.url";
+
/**
* carbon data file version property
*/
+ @CarbonProperty
public static final String CARBON_DATA_FILE_VERSION =
"carbon.data.file.version";
+
@CarbonProperty
public static final String SPARK_SCHEMA_STRING_LENGTH_THRESHOLD =
"spark.sql.sources.schemaStringLengthThreshold";
public static final int SPARK_SCHEMA_STRING_LENGTH_THRESHOLD_DEFAULT =
4000;
+ //TODO: add annotation and whether change to dynamic
--- End diff --
remove it
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230617938
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -117,77 +126,106 @@ private CarbonCommonConstants() {
* CARBON_TIMESTAMP
*/
public static final String CARBON_TIMESTAMP_MILLIS = "dd-MM-
HH:mm:ss:SSS";
+
/**
- * compressor for writing/reading carbondata file
+ * Property for specifying the format of DATE data type column.
+ * e.g. /MM/dd , or using default value
*/
+ @CarbonProperty
+ public static final String CARBON_DATE_FORMAT = "carbon.date.format";
+
+ /**
+ * default value
+ */
+ public static final String CARBON_DATE_DEFAULT_FORMAT = "-MM-dd";
+
+ /**
+ * compressor for writing/reading CarbonData file
+ */
+ @CarbonProperty
public static final String COMPRESSOR = "carbon.column.compressor";
/**
* default compressor is snappy
*/
public static final String DEFAULT_COMPRESSOR = "snappy";
+
/**
- * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper will be
used to handle locking
+ * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper
+ * will be used to handle locking
* mechanism of carbon
*/
@CarbonProperty
public static final String LOCK_TYPE = "carbon.lock.type";
+ /**
+ * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
+ */
+ public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+
/**
* Specifies the path where the lock files have to be created.
* By default, lock files are created in table path.
*/
@CarbonProperty
public static final String LOCK_PATH = "carbon.lock.path";
- /**
- * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
- */
- public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+ public static final String LOCK_PATH_DEFAULT = "";
/**
* ZOOKEEPER_LOCATION this is the location in zookeeper file system
where locks are created.
* mechanism of carbon
*/
public static final String ZOOKEEPER_LOCATION = "/CarbonLocks";
+
/**
- * xxhash algorithm property for hashmap
+ * xxhash algorithm property for hashMap
*/
+ @CarbonProperty
public static final String ENABLE_XXHASH = "carbon.enableXXHash";
/**
- * xxhash algorithm property for hashmap. Default value false
+ * xxhash algorithm property for hashMap Default value false
--- End diff --
keep hashmap
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230617917
--- Diff:
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
---
@@ -117,77 +126,106 @@ private CarbonCommonConstants() {
* CARBON_TIMESTAMP
*/
public static final String CARBON_TIMESTAMP_MILLIS = "dd-MM-
HH:mm:ss:SSS";
+
/**
- * compressor for writing/reading carbondata file
+ * Property for specifying the format of DATE data type column.
+ * e.g. /MM/dd , or using default value
*/
+ @CarbonProperty
+ public static final String CARBON_DATE_FORMAT = "carbon.date.format";
+
+ /**
+ * default value
+ */
+ public static final String CARBON_DATE_DEFAULT_FORMAT = "-MM-dd";
+
+ /**
+ * compressor for writing/reading CarbonData file
+ */
+ @CarbonProperty
public static final String COMPRESSOR = "carbon.column.compressor";
/**
* default compressor is snappy
*/
public static final String DEFAULT_COMPRESSOR = "snappy";
+
/**
- * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper will be
used to handle locking
+ * ZOOKEEPER_ENABLE_LOCK if this is set to true then zookeeper
+ * will be used to handle locking
* mechanism of carbon
*/
@CarbonProperty
public static final String LOCK_TYPE = "carbon.lock.type";
+ /**
+ * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
+ */
+ public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+
/**
* Specifies the path where the lock files have to be created.
* By default, lock files are created in table path.
*/
@CarbonProperty
public static final String LOCK_PATH = "carbon.lock.path";
- /**
- * ZOOKEEPER_ENABLE_DEFAULT the default value for zookeeper will be true
for carbon
- */
- public static final String LOCK_TYPE_DEFAULT = "LOCALLOCK";
+ public static final String LOCK_PATH_DEFAULT = "";
/**
* ZOOKEEPER_LOCATION this is the location in zookeeper file system
where locks are created.
* mechanism of carbon
*/
public static final String ZOOKEEPER_LOCATION = "/CarbonLocks";
+
/**
- * xxhash algorithm property for hashmap
+ * xxhash algorithm property for hashMap
--- End diff --
keep `hashmap`, and can you add comment to describe in where this is used
for? query/loading/compaction?
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230616881
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/path/HDFSLeaseUtils.java ---
@@ -164,22 +157,25 @@ private static boolean recoverLeaseOnFile(String
filePath, Path path, Distribute
private static int getLeaseRecoveryRetryCount() {
String retryMaxAttempts = CarbonProperties.getInstance()
.getProperty(CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT,
-CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
+
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
int retryCount = 0;
try {
retryCount = Integer.parseInt(retryMaxAttempts);
- if (retryCount < CARBON_LEASE_RECOVERY_RETRY_COUNT_MIN
- || retryCount > CARBON_LEASE_RECOVERY_RETRY_COUNT_MAX) {
-retryCount =
Integer.parseInt(CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
+ if (retryCount <
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_MIN
+ || retryCount >
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_MAX) {
+retryCount = Integer.parseInt(
+
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT_DEFAULT);
LOGGER.warn(
"value configured for " +
CarbonCommonConstants.CARBON_LEASE_RECOVERY_RETRY_COUNT
--- End diff --
change this log to use String.format to avoid string concatenation
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230616699
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -722,17 +723,19 @@ private void loadProperties() {
}
} catch (FileNotFoundException e) {
LOGGER.error(
- "The file: " + CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH
+ " does not exist");
+ "The file: " +
CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH_DEFAULT
+ + " does not exist");
} catch (IOException e) {
LOGGER.error(
- "Error while reading the file: " +
CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH);
+ "Error while reading the file: "
+ + CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH_DEFAULT);
--- End diff --
here should use file.getAbsolutePath directly
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2875#discussion_r230616611
--- Diff:
core/src/main/java/org/apache/carbondata/core/util/CarbonProperties.java ---
@@ -722,17 +723,19 @@ private void loadProperties() {
}
} catch (FileNotFoundException e) {
LOGGER.error(
- "The file: " + CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH
+ " does not exist");
+ "The file: " +
CarbonCommonConstants.CARBON_PROPERTIES_FILE_PATH_DEFAULT
--- End diff --
here should use `file.getAbsolutePath` directly
---
[GitHub] carbondata pull request #2875: [CARBONDATA-3038] Refactor dynamic configurat...
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/2875 [CARBONDATA-3038] Refactor dynamic configuration 1.remove CarbonProperty annotation, and DynamicConfigurable, NonDynamicConfigurable, DefaultValue,Threshold and so on 2.unify default value variable name: XX_XX_DEFAULT 3.optimize the properties: remove some warning Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? change the doc - [ ] Testing done no - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. no You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata CARBONDATA-3038_RefactorConfiguration Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2875.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2875 commit 436746f923f90d9db23c6a53a30e39d97a88042d Author: xubo245 Date: 2018-10-29T11:09:15Z [CARBONDATA-3038] Refactor dynamic configuration 1.remove CarbonProperty annotation, and DynamicConfigurable, NonDynamicConfigurable, DefaultValue,Threshold and so on 2.unify default value variable name: XX_XX_DEFAULT 3.optimize the properties: remove some warning optmize ---
