[GitHub] carbondata pull request #2487: [CARBONDATA-2735]Fixed Performance issue for ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2487#discussion_r203608964
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java
---
@@ -175,24 +182,25 @@ private static ColumnPage
getLVBytesColumnPage(TableSpec.ColumnSpec columnSpec,
throws MemoryException {
// extract length and data, set them to rowOffset and unsafe memory
correspondingly
int rowId = 0;
-List rowOffset = new ArrayList<>();
-List rowLength = new ArrayList<>();
+TableSpec.ColumnSpec spec = TableSpec.ColumnSpec
+.newInstance(columnSpec.getFieldName(), DataTypes.INT,
ColumnType.MEASURE);
+ColumnPage rowOffset = ColumnPage.newPage(spec, DataTypes.INT, 1024);
--- End diff --
ok
---
[GitHub] carbondata pull request #2487: [CARBONDATA-2735]Fixed Performance issue for ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2487#discussion_r203608948
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java
---
@@ -201,31 +209,32 @@ private static ColumnPage
getComplexLVBytesColumnPage(TableSpec.ColumnSpec colum
throws MemoryException {
// extract length and data, set them to rowOffset and unsafe memory
correspondingly
int rowId = 0;
-List rowOffset = new ArrayList<>();
-List rowLength = new ArrayList<>();
+TableSpec.ColumnSpec spec = TableSpec.ColumnSpec
+.newInstance(columnSpec.getFieldName(), DataTypes.INT,
ColumnType.MEASURE);
+ColumnPage rowOffset = ColumnPage.newPage(spec, DataTypes.INT, 1024);
--- End diff --
ok
---
[GitHub] carbondata pull request #2487: [CARBONDATA-2735]Fixed Performance issue for ...
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2487#discussion_r203608929
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/page/SafeVarLengthColumnPage.java
---
@@ -40,6 +40,7 @@
@Override
public void freeMemory() {
byteArrayData = null;
+rowOffset.freeMemory();
--- End diff --
ok
---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6063/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2513 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5914/ ---
[GitHub] carbondata pull request #2521: [CARBONDATA-2751] Fixed Thread leak issue in ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2521#discussion_r203603034
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/steps/DataWriterProcessorStepImpl.java
---
@@ -80,6 +85,7 @@ public
DataWriterProcessorStepImpl(CarbonDataLoadConfiguration configuration) {
@Override public void initialize() throws IOException {
super.initialize();
child.initialize();
+this.carbonFactHandlers = new CopyOnWriteArrayList<>();
--- End diff --
Please do similar handing for `DataWriterBatchProcessorStepImpl` as well
---
[GitHub] carbondata pull request #2521: [CARBONDATA-2751] Fixed Thread leak issue in ...
Github user ravipesala commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2521#discussion_r203602958 --- Diff: processing/src/main/java/org/apache/carbondata/processing/loading/steps/DataWriterProcessorStepImpl.java --- @@ -184,12 +190,14 @@ private void processRange(Iterator insideRangeIterator, dataHandler = CarbonFactHandlerFactory .createCarbonFactHandler(model); dataHandler.initialise(); +carbonFactHandlers.add(dataHandler); --- End diff -- Better add before initialize, so can you move this line above. ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7296/ ---
[jira] [Created] (CARBONDATA-2754) fix failing UT for HiveMetastore
Rahul Kumar created CARBONDATA-2754: --- Summary: fix failing UT for HiveMetastore Key: CARBONDATA-2754 URL: https://issues.apache.org/jira/browse/CARBONDATA-2754 Project: CarbonData Issue Type: Improvement Reporter: Rahul Kumar Assignee: Rahul Kumar -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2522 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6064/ ---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2522 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7297/ ---
[GitHub] carbondata issue #2503: [CARBONDATA-2734] Update is not working on the table...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2503 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5913/ ---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2522 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7295/ ---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Github user chenliang613 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2522#discussion_r203592254 --- Diff: zeppelin/README.txt --- @@ -0,0 +1,18 @@ +Please follow below steps to integrate with zeppelin --- End diff -- 1. please use .md format 2. Please add apache license header also. ---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2522 add to whitelist ---
[GitHub] carbondata issue #2511: [CARBONDATA-2745] Added atomic file operations for S...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2511 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5912/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2513 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5911/ ---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2484 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5910/ ---
[GitHub] carbondata issue #2521: [CARBONDATA-2751] Fixed Thread leak issue in data lo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2521 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5909/ ---
[GitHub] carbondata issue #2523: [CARBONDATA-2753] Fix Compatibility issue with Preag...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2523 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6061/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6060/ ---
[GitHub] carbondata issue #2487: [CARBONDATA-2735]Fixed Performance issue for complex...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2487 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5908/ ---
[GitHub] carbondata issue #2523: [CARBONDATA-2753] Fix Compatibility issue with Preag...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2523 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7294/ ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2415 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6058/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7293/ ---
[GitHub] carbondata issue #2516: [WIP][Complex DataType Enhancements] Complex compact...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2516 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5907/ ---
[GitHub] carbondata issue #2523: [CARBONDATA-2753] Fix Compatibility issue with Preag...
Github user Indhumathi27 commented on the issue: https://github.com/apache/carbondata/pull/2523 Retest this please ---
[GitHub] carbondata issue #2523: [CARBONDATA-2753] Fix Compatibility issue with Preag...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2523 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7292/ ---
[GitHub] carbondata issue #2474: [CARBONDATA-2530][MV] Disable the MV datamaps after ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2474 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6057/ ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2415 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7291/ ---
[GitHub] carbondata issue #2412: [CARBONDATA-2656] Presto vector stream readers perfo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2412 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5906/ ---
[GitHub] carbondata issue #2478: [CARBONDATA-2540][CARBONDATA-2560][CARBONDATA-2568][...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2478 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6056/ ---
[GitHub] carbondata issue #2474: [CARBONDATA-2530][MV] Disable the MV datamaps after ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2474 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7290/ ---
[GitHub] carbondata issue #2480: [CARBONDATA-2550][CARBONDATA-2576][MV] Fix limit and...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2480 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6055/ ---
[GitHub] carbondata issue #2478: [CARBONDATA-2540][CARBONDATA-2560][CARBONDATA-2568][...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2478 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7289/ ---
[GitHub] carbondata issue #2480: [CARBONDATA-2550][CARBONDATA-2576][MV] Fix limit and...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2480 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7288/ ---
[GitHub] carbondata issue #2520: [CARBONDATA-2750] Added Documentation for Local Dict...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2520 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5905/ ---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6054/ ---
[GitHub] carbondata issue #2511: [CARBONDATA-2745] Added atomic file operations for S...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2511 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6053/ ---
[GitHub] carbondata pull request #2523: [CARBONDATA-2753] Fix Compatibility issue wit...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2523 [CARBONDATA-2753] Fix Compatibility issue with Preaggregate table Problem: User creates maintable and load data and create preaggregate datamap on maintable using old version and loads data into maintable using new version and dataload fails. - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata preagg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2523.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2523 commit 7bfd652d3c1513206b0e5d719e68caccea6c2d47 Author: Indhumathi27 Date: 2018-07-18T15:17:06Z [CARBONDATA-2753] Fix Compatibility issue with Preaggregate table ---
[jira] [Created] (CARBONDATA-2753) Fix Compatibility issue with Preaggregate
Indhumathi Muthumurugesh created CARBONDATA-2753: Summary: Fix Compatibility issue with Preaggregate Key: CARBONDATA-2753 URL: https://issues.apache.org/jira/browse/CARBONDATA-2753 Project: CarbonData Issue Type: Bug Reporter: Indhumathi Muthumurugesh Assignee: Indhumathi Muthumurugesh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2511: [CARBONDATA-2745] Added atomic file operations for S...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2511 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7286/ ---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7287/ ---
[GitHub] carbondata pull request #2513: [CARBONDATA-2748] blocking concurrent load if...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2513#discussion_r203414767
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -253,15 +257,39 @@ case class CarbonLoadDataCommand(
}
// First system has to partition the data first and then call the
load data
LOGGER.info(s"Initiating Direct Load for the Table :
($dbName.$tableName)")
-// Clean up the old invalid segment data before creating a new
entry for new load.
-SegmentStatusManager.deleteLoadsAndUpdateMetadata(table, false,
currPartitions)
-// add the start entry for the new load in the table status file
-if (updateModel.isEmpty && !table.isHivePartitionTable) {
- CarbonLoaderUtil.readAndUpdateLoadProgressInTableMeta(
-carbonLoadModel,
-isOverwriteTable)
- isUpdateTableStatusRequired = true
+
--- End diff --
Add function to acquire and release concurrent lock.
---
[GitHub] carbondata pull request #2513: [CARBONDATA-2748] blocking concurrent load if...
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2513#discussion_r203414480
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
---
@@ -355,6 +383,14 @@ case class CarbonLoadDataCommand(
val file = FileFactory.getCarbonFile(partitionLocation,
fileType)
CarbonUtil.deleteFoldersAndFiles(file)
}
+ if (isConcurrentLockRequired && !concurrentLoadLock.unlock()) {
+LOGGER
+ .info("concurrent_load lock for table" + table.getTablePath +
+"has been released successfully")
+ } else {
+LOGGER.error(
+ "Unable to unlock concurrent_load lock for table" +
table.getTablePath);
+ }
--- End diff --
Unlocking should be in finally.
---
[GitHub] carbondata pull request #2517: [CARBONDATA-2749][dataload] In HDFS Empty tab...
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2517#discussion_r203411199
--- Diff:
core/src/main/java/org/apache/carbondata/core/fileoperations/AtomicFileOperationsImpl.java
---
@@ -70,12 +78,20 @@ public AtomicFileOperationsImpl(String filePath,
FileType fileType) {
if (null != dataOutStream) {
CarbonUtil.closeStream(dataOutStream);
CarbonFile tempFile = FileFactory.getCarbonFile(tempWriteFilePath,
fileType);
- if (!tempFile.renameForce(filePath)) {
-throw new IOException("temporary file renaming failed, src="
-+ tempFile.getPath() + ", dest=" + filePath);
+ if (!this.setFailed) {
+if (!tempFile.renameForce(filePath)) {
+ throw new IOException(
+ "temporary file renaming failed, src=" + tempFile.getPath()
+ ", dest=" + filePath);
+}
}
+} else {
+ LOGGER.warn("The temporary file renaming skipped due to I/O error,
deleting file "
+ + tempWriteFilePath);
}
}
+ @Override public void setFailed() {
--- End diff --
setFailed needs to be raised on Exception cases in the below usages of
AtomicFileOperations
-
org.apache.carbondata.core.datamap.status.DiskBasedDataMapStatusProvider#writeLoadDetailsIntoFile
-
org.apache.carbondata.core.metadata.SegmentFileStore#writeSegmentFile(org.apache.carbondata.core.metadata.SegmentFileStore.SegmentFile,
java.lang.String)
-
org.apache.carbondata.core.statusmanager.SegmentStatusManager#writeLoadMetadata
Please also handle any other references I missed.
---
[GitHub] carbondata issue #2387: [CARBONDATA-2621][BloomDataMap] Lock problem in inde...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2387 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5904/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6052/ ---
[GitHub] carbondata issue #2503: [CARBONDATA-2734] Update is not working on the table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2503 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6051/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7285/ ---
[GitHub] carbondata issue #2503: [CARBONDATA-2734] Update is not working on the table...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2503 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7284/ ---
[GitHub] carbondata issue #2415: [CARBONDATA-2659] Support partition table by DataFra...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2415 retest this please ---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2484 retest this please ---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Github user mohammadshahidkhan commented on the issue: https://github.com/apache/carbondata/pull/2517 The failled test case is unrelated and random. SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5903/ ---
[GitHub] carbondata issue #2517: [CARBONDATA-2749][dataload] In HDFS Empty tablestatu...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2517 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5903/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6049/ ---
[GitHub] carbondata issue #2513: [CARBONDATA-2748] blocking concurrent load if any co...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2513 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7282/ ---
[GitHub] carbondata issue #2511: [CARBONDATA-2745] Added atomic file operations for S...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2511 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6050/ ---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6048/ ---
[GitHub] carbondata issue #2511: [CARBONDATA-2745] Added atomic file operations for S...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2511 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7283/ ---
[jira] [Resolved] (CARBONDATA-2741) Exception occurs after alter add few columns and selecting in random order
[
https://issues.apache.org/jira/browse/CARBONDATA-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravindra Pesala resolved CARBONDATA-2741.
-
Resolution: Fixed
Fix Version/s: (was: 1.5.0)
1.4.1
> Exception occurs after alter add few columns and selecting in random order
> --
>
> Key: CARBONDATA-2741
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2741
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
>Affects Versions: 1.5.0
> Environment: 3 node cluster with spark2.2
>Reporter: Jatin
>Assignee: Jatin
>Priority: Minor
> Fix For: 1.4.1
>
>
> create table tb1 (imei string,AMSize string,channelsId string,ActiveCountry
> string, Activecity string,gamePointId double,deviceInformationId
> double,productionDate Timestamp,deliveryDate timestamp,deliverycharge double)
> STORED BY 'org.apache.carbondata.format'
> TBLPROPERTIES('table_blocksize'='1','COLUMN_META_CACHE'='AMSize');
> LOAD DATA INPATH 'hdfs://hacluster/csv/vardhandaterestruct.csv' INTO TABLE
> tb1 OPTIONS('DELIMITER'=',', 'QUOTECHAR'=
> '"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'=
> 'imei,deviceInformationId,AMSize,channelsId,ActiveCountry,Activecity,gamePointId,productionDate,deliveryDate,deliverycharge');
> alter table tb1 add columns(age int, name string);
> select * from tb1 where name is NULL or channelsId =4;
> Exception occurs :
> *Error: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task 0 in stage 6508.0 failed 4 times, most recent failure: Lost task 0.3 in
> stage 6508.0 (TID 140476, linux-49, executor 3): java.lang.RuntimeException:
> internal error:*
> org.apache.carbondata.core.datastore.page.encoding.adaptive.AdaptiveFloatingCodec[src
> type: DOUBLE, target type: INT, stats(min: 1.0, max: 100.0, decimal: 1 )]
> at
> org.apache.carbondata.core.datastore.page.encoding.adaptive.AdaptiveFloatingCodec$3.decodeLong(AdaptiveFloatingCodec.java:185)
> at
> org.apache.carbondata.core.datastore.page.LazyColumnPage.getLong(LazyColumnPage.java:64)
> at
> org.apache.carbondata.core.scan.result.vector.MeasureDataVectorProcessor$IntegralMeasureVectorFiller.fillMeasureVector(MeasureDataVectorProcessor.java:73)
> at
> org.apache.carbondata.core.scan.result.impl.FilterQueryScannedResult.fillColumnarMeasureBatch(FilterQueryScannedResult.java:129)
> at
> org.apache.carbondata.core.scan.collector.impl.DictionaryBasedVectorResultCollector.fillResultToColumnarBatch(DictionaryBasedVectorResultCollector.java:167)
> at
> org.apache.carbondata.core.scan.collector.impl.RestructureBasedVectorResultCollector.collectResultInColumnarBatch(RestructureBasedVectorResultCollector.java:127)
> at
> org.apache.carbondata.core.scan.processor.DataBlockIterator.processNextBatch(DataBlockIterator.java:245)
> at
> org.apache.carbondata.core.scan.result.iterator.VectorDetailQueryResultIterator.processNextBatch(VectorDetailQueryResultIterator.java:48)
> at
> org.apache.carbondata.spark.vectorreader.VectorizedCarbonRecordReader.nextBatch(VectorizedCarbonRecordReader.java:290)
> at
> org.apache.carbondata.spark.vectorreader.VectorizedCarbonRecordReader.nextKeyValue(VectorizedCarbonRecordReader.java:180)
> at
> org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.hasNext(CarbonScanRDD.scala:497)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:381)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:381)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:828)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:828)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
> at
[GitHub] carbondata pull request #2507: [CABONDATA-2741]Fix for filling measure colum...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2507 ---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347890
--- Diff:
store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java
---
@@ -42,20 +43,24 @@
public ResponseEntity sql(@RequestBody SqlRequest request)
throws StoreException {
RequestValidator.validateSql(request);
List rows;
+Dataset sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(),
request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
-Object[][] result = new Object[rows.size()][];
+Object[][] result = new Object[rows.size()+1][];
+result[0] = sqlDataFrame.schema().fieldNames();
for (int i = 0; i < rows.size(); i++) {
--- End diff --
`I ` can start from 1
---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7281/ ---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347726
--- Diff:
store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java
---
@@ -42,20 +43,24 @@
public ResponseEntity sql(@RequestBody SqlRequest request)
throws StoreException {
RequestValidator.validateSql(request);
List rows;
+Dataset sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(),
request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
-Object[][] result = new Object[rows.size()][];
+Object[][] result = new Object[rows.size()+1][];
+result[0] = sqlDataFrame.schema().fieldNames();
for (int i = 0; i < rows.size(); i++) {
Row row = rows.get(i);
- result[i] = new Object[row.size()];
+ result[i+1] = new Object[row.size()];
for (int j = 0; j < row.size(); j++) {
-result[i][j] = row.get(j);
+result[i+1][j] = row.get(j);
--- End diff --
Can't use System.arraycopy?
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347064
--- Diff:
store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java
---
@@ -42,20 +43,24 @@
public ResponseEntity sql(@RequestBody SqlRequest request)
throws StoreException {
RequestValidator.validateSql(request);
List rows;
+Dataset sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(),
request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
--- End diff --
move it to previous line
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347146
--- Diff:
store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java
---
@@ -42,20 +43,24 @@
public ResponseEntity sql(@RequestBody SqlRequest request)
throws StoreException {
RequestValidator.validateSql(request);
List rows;
+Dataset sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(),
request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
-Object[][] result = new Object[rows.size()][];
+Object[][] result = new Object[rows.size()+1][];
--- End diff --
add space before and after `+`
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2522 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2522 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2521: [CARBONDATA-2751] Fixed Thread leak issue in data lo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2521 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6047/ ---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2522 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2518: [WIP] fixing testcases if HiveMetastore is enabled
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2518 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5902/ ---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
GitHub user ajithme opened a pull request: https://github.com/apache/carbondata/pull/2522 [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppelin support Apache Zeppelin is a popular open web-based notebook that enables interactive data analytics. This is one of the favored solutions for providing UI frontend as it can support solutions like Spark already. Carbon can leverage this to provide a UI for its operations. After CARBONDATA-2688 which provides a carbon REST server, we can add a UI support from zeppelin to provide a complete solution. - [ ] Document update required? YES, need to update usage guide for Zeppelin integration - [ ] Testing done 1. Added UT 2. Done Testing manually by integration with Zeppelin You can merge this pull request into a Git repository by running: $ git pull https://github.com/ajithme/carbondata zeppelinsupport Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2522.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2522 commit 16c69836c3903150875582f950f20cd1189fc69a Author: Ajith Date: 2018-07-18T11:18:54Z CARBONDATA-2752 Zeppelin support commit 13801259e5c46b1a4cc736fe94c4bf3678d75794 Author: Ajith Date: 2018-07-18T11:26:29Z update doc ---
[jira] [Resolved] (CARBONDATA-2710) Refactor CarbonSparkSqlParser for better code reuse.
[ https://issues.apache.org/jira/browse/CARBONDATA-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta resolved CARBONDATA-2710. -- Resolution: Fixed Assignee: Mohammad Shahid Khan Fix Version/s: 1.4.1 > Refactor CarbonSparkSqlParser for better code reuse. > > > Key: CARBONDATA-2710 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2710 > Project: CarbonData > Issue Type: Improvement >Reporter: Mohammad Shahid Khan >Assignee: Mohammad Shahid Khan >Priority: Major > Fix For: 1.4.1 > > Time Spent: 4.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2466: [CARBONDATA-2710][Spark Integration] Refactor...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2466 ---
[GitHub] carbondata issue #2521: [CARBONDATA-2751] Fixed Thread leak issue in data lo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2521 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7280/ ---
[GitHub] carbondata issue #2516: [WIP][Complex DataType Enhancements] Complex compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2516 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6046/ ---
[jira] [Commented] (CARBONDATA-2736) Support Kafka Source for CREATE STREAM
[ https://issues.apache.org/jira/browse/CARBONDATA-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547680#comment-16547680 ] Ajith S commented on CARBONDATA-2736: - Merged https://github.com/apache/carbondata/commit/9ac55a5a656ebe106697ca76a04916bea2ef3109 > Support Kafka Source for CREATE STREAM > -- > > Key: CARBONDATA-2736 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2736 > Project: CarbonData > Issue Type: Bug >Reporter: Ajith S >Assignee: Ajith S >Priority: Major > Attachments: image-2018-07-12-11-58-49-346.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When CREATE STREAM over a kafka source. It will throw a > IllegalArgumentException > > !image-2018-07-12-11-58-49-346.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2495: [CARBONDATA-2736][CARBONSTORE] Kafka integration wit...
Github user ajithme commented on the issue: https://github.com/apache/carbondata/pull/2495 Merged https://github.com/apache/carbondata/commit/9ac55a5a656ebe106697ca76a04916bea2ef3109 ---
[GitHub] carbondata pull request #2495: [CARBONDATA-2736][CARBONSTORE] Kafka integrat...
Github user ajithme closed the pull request at: https://github.com/apache/carbondata/pull/2495 ---
[jira] [Assigned] (CARBONDATA-2736) Support Kafka Source for CREATE STREAM
[ https://issues.apache.org/jira/browse/CARBONDATA-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S reassigned CARBONDATA-2736: --- Assignee: Ajith S > Support Kafka Source for CREATE STREAM > -- > > Key: CARBONDATA-2736 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2736 > Project: CarbonData > Issue Type: Bug >Reporter: Ajith S >Assignee: Ajith S >Priority: Major > Attachments: image-2018-07-12-11-58-49-346.png > > Time Spent: 1h > Remaining Estimate: 0h > > When CREATE STREAM over a kafka source. It will throw a > IllegalArgumentException > > !image-2018-07-12-11-58-49-346.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2387: [CARBONDATA-2621][BloomDataMap] Lock problem in inde...
Github user mohammadshahidkhan commented on the issue: https://github.com/apache/carbondata/pull/2387 Failled test case for the below build is unrelated. The same testcase is passing locally. http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7258/ ---
[GitHub] carbondata issue #2466: [CARBONDATA-2710][Spark Integration] Refactor Carbon...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2466 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5901/ ---
[GitHub] carbondata issue #2516: [WIP][Complex DataType Enhancements] Complex compact...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2516 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7277/ ---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6045/ ---
[jira] [Created] (CARBONDATA-2752) Carbon provide Zeppelin support
Ajith S created CARBONDATA-2752: --- Summary: Carbon provide Zeppelin support Key: CARBONDATA-2752 URL: https://issues.apache.org/jira/browse/CARBONDATA-2752 Project: CarbonData Issue Type: New Feature Reporter: Ajith S Assignee: Ajith S Fix For: 1.5.0 Attachments: image-2018-07-18-17-09-04-583.png *Apache Zeppelin* is a popular open web-based notebook that enables interactive data analytics. This is one of the favored solutions for providing UI frontend as it can support solutions like Spark already. Carbon can leverage this to provide a UI for its operations. After CARBONDATA-2688 which provides a carbon REST server, we can add a UI support from zeppelin to provide a complete solution. Reference: [https://zeppelin.apache.org/] +Proposed solution:+ !image-2018-07-18-17-09-04-583.png! This JIRA propose to add a carbon based interpreter for Zeppelin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (CARBONDATA-2751) Thread leak issue in data loading and Compatibility issue
kumar vishal created CARBONDATA-2751: Summary: Thread leak issue in data loading and Compatibility issue Key: CARBONDATA-2751 URL: https://issues.apache.org/jira/browse/CARBONDATA-2751 Project: CarbonData Issue Type: Bug Reporter: kumar vishal Assignee: kumar vishal *### Problem:* * Thread leak when user is killing data loading process from UI * NPE when user is querying old store. *### Solution* * When carbondata file writing is in progress during data loading and user is killing it from UI Producer and consumer thread are not getting shutdown. Need to handle the same in close method * Old store (V1/V2) does not have datachunk3 object so while filling the local dictionary it is checking whether local dictionary is present in datachunk3 or not but datachunk3 null check is missing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2520: [CARBONDATA-2750] Added Documentation for Local Dict...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2520 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6044/ ---
[GitHub] carbondata issue #2412: [CARBONDATA-2656] Presto vector stream readers perfo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2412 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6043/ ---
[GitHub] carbondata issue #2519: [CARBONDATA-2747][Lucene] Fix Lucene datamap choosin...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2519 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5900/ ---
[GitHub] carbondata issue #2484: [HOTFIX] added hadoop conf to thread local
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2484 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7276/ ---
[GitHub] carbondata issue #2520: [CARBONDATA-2750] Added Documentation for Local Dict...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2520 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7275/ ---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
Github user sgururajshetty commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2520#discussion_r203281996
--- Diff: docs/data-management-on-carbondata.md ---
@@ -291,6 +330,11 @@ This tutorial is going to introduce all commands and
data operations on CarbonDa
ALTER TABLE carbon ADD COLUMNS (a1 INT, b1 STRING)
TBLPROPERTIES('DEFAULT.VALUE.a1'='10')
```
+ Users can specify which columns to include and exclude for local
dictionary generation after adding new columns. These will be appended with
the already existing local dictionary include and exclude columns of
main table respectively.
--- End diff --
check the spacing between words
---
[GitHub] carbondata issue #2412: [CARBONDATA-2656] Presto vector stream readers perfo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2412 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7274/ ---
[GitHub] carbondata issue #2487: [CARBONDATA-2735]Fixed Performance issue for complex...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2487 retest sdv please ---
[GitHub] carbondata pull request #2521: [WIP] Fixed Thread leak issue in data loading
GitHub user kumarvishal09 opened a pull request: https://github.com/apache/carbondata/pull/2521 [WIP] Fixed Thread leak issue in data loading Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kumarvishal09/incubator-carbondata threadleakfixNew Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2521.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2521 commit 58c51e37c08adc782b5e503a452531e4997351f4 Author: kumarvishal09 Date: 2018-07-18T07:36:36Z Fixed Thread Leak Issue commit 89bf7ec06b2478efd20bd8f03bf59f3c988c1c44 Author: kumarvishal09 Date: 2018-07-18T07:39:47Z Fixed Thread Leak Issue ---
[GitHub] carbondata pull request #2520: [CARBONDATA-2750] Added Documentation for Loc...
Github user sgururajshetty commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2520#discussion_r203277284
--- Diff: docs/data-management-on-carbondata.md ---
@@ -122,6 +122,45 @@ This tutorial is going to introduce all commands and
data operations on CarbonDa
TBLPROPERTIES ('streaming'='true')
```
+ - **Local Dictionary Configuration**
+
+ Local Dictionary is generated only for no-dictionary string/varchar
datatype columns. It helps in:
+ 1. Getting more compression on dimension columns with less cardinality.
+ 2. Filter queries and full scan queries on No-dictionary columns with
local dictionary will be faster as filter will be done on encoded data.
+ 3. Reducing the store size and memory footprint as only unique values
will be stored as part of local dictionary and corresponding data will be
stored as encoded data.
+
+ By default, Local Dictionary will be enabled and generated for all
no-dictionary string/varchar datatype columns.
--- End diff --
Convert this into table
| Properties | Default Value | Description |
The **description** should satisfy the following points:
a. What does this parameter do?
b. In what scenario the user needs to configure this parameter?
c. Are there any benefits in configuring this parameter?
d. What is the default value?
e. What is the value range if any?
f. Are there any limitations?
g. Any key information to be highlighted?
---
[GitHub] carbondata issue #2518: [WIP] fixing testcases if HiveMetastore is enabled
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2518 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6042/ ---
[GitHub] carbondata issue #2519: [CARBONDATA-2747][Lucene] Fix Lucene datamap choosin...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2519 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6041/ ---
[GitHub] carbondata issue #2412: [CARBONDATA-2656] Presto vector stream readers perfo...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2412 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5899/ ---
