[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user xubo245 closed the pull request at: https://github.com/apache/carbondata/pull/2836 ---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2836#discussion_r229284826 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1234,7 +1234,7 @@ @CarbonProperty public static final String UNSAFE_WORKING_MEMORY_IN_MB = "carbon.unsafe.working.memory.in.mb"; - public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "512"; + public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "1024"; --- End diff -- You can change the configuration in your application, there is no need to change the default value of this parameter ---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2836#discussion_r229188244
--- Diff: store/sdk/src/main/resources/log4j.properties ---
@@ -0,0 +1,11 @@
+# Root logger option
+log4j.rootLogger=INFO,stdout
+
+
+# Redirect log messages to console
+log4j.appender.debug=org.apache.log4j.RollingFileAppender
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.Target=System.out
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=%d{-MM-dd HH:mm:ss}
%-5p %c{1}:%L - %m%n
--- End diff --
there is already add %c{1} to print class name:
```
2018-10-30 14:26:23 INFO CacheProvider:168 - Executor LRU cache size not
configured. Initializing with driver LRU cache size.
2018-10-30 14:26:23 INFO CarbonLRUCache:76 - LRU cache size not
configured. Therefore default behavior will be considered and no LRU based
eviction of columns will be done
2018-10-30 14:26:23 INFO SegmentPropertiesAndSchemaHolder:117 -
Constructing new SegmentProperties for table: null_null. Current size of
segment properties holder list is: 1
2018-10-30 14:26:23 INFO QueryModelBuilder:94 - Projection Columns:
[stringfield, datefield, timefield, varcharfield, arrayfield, shortfield,
intfield, longfield, doublefield, boolfield, decimalfield, floatfield]
2018-10-30 14:26:23 INFO AbstractQueryExecutor:121 - Query will be
executed on table: null
2018-10-30 14:26:23 INFO ResultCollectorFactory:79 - Row based dictionary
collector is used to scan and collect the data
2018-10-30 14:26:24 WARN DataMapStoreManager:499 - failed to get carbon
table from table PathFile does not exist:
/Users/xubo/Desktop/xubo/git/carbondata1/store/sdk/carbondata/Metadata/schema
2018-10-30 14:26:24 INFO CarbonLRUCache:163 - Removed entry from InMemory
lru cache ::
/Users/xubo/Desktop/xubo/git/carbondata1/store/sdk/carbondata/632392937014916_batchno0-0-null-632392085265031.carbonindex
2018-10-30 14:26:24 INFO UnsafeMemoryManager:175 - Total offheap working
memory used after task 14e4fd01-9df5-416a-9b0a-6970ea63c2bd is 0. Current
running tasks are d26d0593-f4ce-4bb8-ac94-974638229901,
a92307f4-9292-4645-87fb-2a772b421df4
```
---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2836#discussion_r229170138 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1234,7 +1234,7 @@ @CarbonProperty public static final String UNSAFE_WORKING_MEMORY_IN_MB = "carbon.unsafe.working.memory.in.mb"; - public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "512"; + public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "1024"; --- End diff -- ok, I described the issue in ahead ---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2836#discussion_r227623050
--- Diff: store/sdk/src/main/resources/log4j.properties ---
@@ -0,0 +1,11 @@
+# Root logger option
+log4j.rootLogger=INFO,stdout
+
+
+# Redirect log messages to console
+log4j.appender.debug=org.apache.log4j.RollingFileAppender
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.Target=System.out
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=%d{-MM-dd HH:mm:ss}
%-5p %c{1}:%L - %m%n
--- End diff --
Now we are using log4j Logger directly, you can add %C also in the pattern
to print the class name
---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2836#discussion_r227622952 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1234,7 +1234,7 @@ @CarbonProperty public static final String UNSAFE_WORKING_MEMORY_IN_MB = "carbon.unsafe.working.memory.in.mb"; - public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "512"; + public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "1024"; --- End diff -- Can you describe the issue you encountered using the original default value? ---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2836#discussion_r226548201 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1234,7 +1234,7 @@ @CarbonProperty public static final String UNSAFE_WORKING_MEMORY_IN_MB = "carbon.unsafe.working.memory.in.mb"; - public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "512"; + public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "2048"; --- End diff -- I don't know. ---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
Github user xuchuanyin commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2836#discussion_r226544482 --- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java --- @@ -1234,7 +1234,7 @@ @CarbonProperty public static final String UNSAFE_WORKING_MEMORY_IN_MB = "carbon.unsafe.working.memory.in.mb"; - public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "512"; + public static final String UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT = "2048"; --- End diff -- What will happen if user do not have that much memory? ---
[GitHub] carbondata pull request #2836: [CARBONDATA-3027] Increase unsafe working mem...
GitHub user xubo245 opened a pull request: https://github.com/apache/carbondata/pull/2836 [CARBONDATA-3027] Increase unsafe working memory default size and add log file for SDK [CARBONDATA-3027] Increase unsafe working memory default size and add log file for SDK 1.Increase unsafe working memory default size from 512m to 2048m 2.add log file for SDK Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? No - [ ] Any backward compatibility impacted? No - [ ] Document update required? No - [ ] Testing done No - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. No You can merge this pull request into a Git repository by running: $ git pull https://github.com/xubo245/carbondata CARBONDATA-3027MemoryException Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2836.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2836 commit 90ea4ff1f73ac04cb035618a70b1458049fa0e86 Author: xubo245 Date: 2018-10-19T03:50:40Z [CARBONDATA-3027] Increase unsafe working memory default size and add log file for SDK 1.Increase unsafe working memory default size from 512m to 2048m 2.add log file for SDK ---
