Github user sgururajshetty commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2756#discussion_r220427375
--- Diff: docs/configuration-parameters.md ---
@@ -42,6 +42,7 @@ This section provides the details of all the
configurations required for the Car
| carbon.lock.type | LOCALLOCK | This configuration specifies the type of
lock to be acquired during concurrent operations on table. There are following
types of lock implementation: - LOCALLOCK: Lock is created on local file system
as file. This lock is useful when only one spark driver (thrift server) runs on
a machine and no other CarbonData spark application is launched concurrently. -
HDFSLOCK: Lock is created on HDFS file system as file. This lock is useful when
multiple CarbonData spark applications are launched and no ZooKeeper is running
on cluster and HDFS supports file based locking. |
| carbon.lock.path | TABLEPATH | This configuration specifies the path
where lock files have to be created. Recommended to configure zookeeper lock
type or configure HDFS lock path(to this property) in case of S3 file system as
locking is not feasible on S3. |
| carbon.unsafe.working.memory.in.mb | 512 | CarbonData supports storing
data in off-heap memory for certain operations during data loading and
query.This helps to avoid the Java GC and thereby improve the overall
performance.The Minimum value recommeded is 512MB.Any value below this is reset
to default value of 512MB.**NOTE:** The below formulas explain how to arrive at
the off-heap size required.<u>Memory Required For Data
Loading:</u>(*carbon.number.of.cores.while.loading*) * (Number of tables to
load in parallel) * (*offheap.sort.chunk.size.inmb* +
*carbon.blockletgroup.size.in.mb* + *carbon.blockletgroup.size.in.mb*/3.5 ).
<u>Memory required for Query:</u>SPARK_EXECUTOR_INSTANCES *
(*carbon.blockletgroup.size.in.mb* + *carbon.blockletgroup.size.in.mb* * 3.5) *
spark.executor.cores |
+| carbon.unsafe.driver.working.memory.in.mb | 60% of JVM Heap Memory |
CarbonData supports storing data in unsafe on-heap memory in driver for certain
operations like insert into, query for loading datamap cache. The Minimum value
recommended is 512MB. |
--- End diff --
Parameter description should satisfy following questions:
a. What does this parameter do?
b. In what scenario the user needs to configure this parameter?
c. Is there any benefits by configuring these parameter?
d. What is the default value?
e. What is the value range if any?
f. Is there any limitations?
g. Any key information to be highlighted?
---
