GitHub user ManoharVanam reopened a pull request:
https://github.com/apache/carbondata/pull/972
[CARBONDATA-1065] Added set command in carbon to update properties
dynamically
**Requirement** : Currently there is no way to set the carbon properties
dynamically, always need to update the carbon.properties file and restart the
driver.
So it will be easy for users to give option to add/update/display/reset the
properties dynamically using set and reset commands.
**Solution :**
The below sql command updates the carbon properties in driver and as well
as in executor
**Add/update/display session property :**
set parameter = value --> add/update carbon session property in the
corresponding session
Example : set enable.unsafe.sort = true;
set parameter --> displays given parameter value in the corresponding
session
Example : set enable.unsafe.sort;
**Reset all session properties :**
Reset --> clears all session parameters in the corresponding session
Example : reset;
**SET:**
SET command displays all session supported parameters (carbon + spark) in
the corresponding session
Example : SET;
**SET -V:**
set -v command displays all session parameter list along with default value
and usage doc
Example : set -v;
**Supported parameters for dynamic set.**
carbon.options.bad.records.logger.enable
carbon.options.bad.records.action
carbon.options.is.empty.data.bad.record
carbon.options.dateformat
carbon.options.sort.scope
carbon.options.batch.sort.size.inmb
carbon.options.single.pass
carbon.options.bad.record.path
carbon.options.global.sort.partitions
enable.unsafe.sort
carbon.custom.block.distribution
Example:
To enable bad records logger using set command
_set carbon.options.bad.records.logger.enable=true_
**Parameter precedence **
1. Load Options
2. Set Command options
3. Global Property
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ManoharVanam/incubator-carbondata dynamic_new
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/972.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #972
----
commit 22554e40ff50e5eb14633b1fe1d8319d278cbecd
Author: ravipesala <[email protected]>
Date: 2017-05-18T09:34:17Z
Adding session based properties
Added set command in carbon to update properties dynamically
commit 11f4dff6a67e50eb2778116a24fc191ebe9ce0c0
Author: Manohar <[email protected]>
Date: 2017-06-12T12:36:25Z
Added set/reset commands in carbon to update/reset properties dynamically
commit 1798a6826a2516676a75e150335ef4659394f2d7
Author: mohammadshahidkhan <[email protected]>
Date: 2017-06-12T13:03:22Z
1. Refactored the bad record code, by default the bad record path will be
empty, if bad record logger is
enabled or action is redirect and bad record path is not configured then
data-load will fail.
2. Support dynamic set command for some of load options
3. fixed test cases
4. Added validation for the supported property in the dynamic set command
5. Change table delete behavior // now the bad record would not be deleted
ion table drop
6. added test case for bad record path in load option
7. fixed failing test cases
8. Added "carbon.options." in load options parameters
commit 3736f71c3b9a6d99b60779befe82ef7caee3c5fc
Author: Manohar <[email protected]>
Date: 2017-06-23T10:50:15Z
Created CarbonSessionInfo wrapper for storing all session details
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
