[GitHub] carbondata issue #2734: [CARBONDATA-2946] Unify conversion while writing to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2734 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/462/ ---
[GitHub] carbondata pull request #2747: [CARBONDATA-2960] SDK Reader fix with project...
Github user manishnalla1994 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2747#discussion_r220062362
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/api/CarbonInputFormat.java ---
@@ -775,9 +775,18 @@ public static boolean
getAccessStreamingSegments(Configuration configuration) {
public String[] projectAllColumns(CarbonTable carbonTable) {
List colList =
carbonTable.getTableInfo().getFactTable().getListOfColumns();
List projectColumn = new ArrayList<>();
+int childDimCount = 0;
for (ColumnSchema cols : colList) {
if (cols.getSchemaOrdinal() != -1) {
-projectColumn.add(cols.getColumnName());
+if (childDimCount == 0) {
--- End diff --
added and updated
---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/461/ ---
[GitHub] carbondata issue #2751: [CARBONDATA-2946] Add bloomindex version info file f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2751 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/640/ ---
[GitHub] carbondata issue #2751: [CARBONDATA-2946] Add bloomindex version info file f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2751 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8710/ ---
[GitHub] carbondata issue #2734: [CARBONDATA-2946] Unify conversion while writing to ...
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2734
Hi, I've tested this PR in local machine and it works fine.
Steps used to verify this:
```
1. Use CarbonData 1.4.1-RC2 jar and start spark & JDBCServer & beeline
2. CREATE TABLE
create table test_adpt_int (id int, name string, age int) stored by
'carbondata' TBLPROPERTIES('sort_columns'='id');
3. CREATE DATAMAP
create datamap dm_id on table test_adpt_int using 'bloomfilter'
DMPROPERTIES('index_columns'='id');
4. LOAD
insert into table test_adpt_int values (1, 'name1', 10),(3, 'name3',
30),(5, 'name5', 50),(7, 'name7', 70),(9, 'name9', 90),(10, 'name10', 100);
5. QUERY
select * from test_adpt_int where id = 6;
select * from test_adpt_int where id = 5;
6. Use master code and apply current PR to generate jar and restart spark &
JDBCServer & beeline
7. QUERY should work fine
select * from test_adpt_int where id = 6;
select * from test_adpt_int where id = 5;
8. LOAD again
insert into table test_adpt_int values (1, 'name1', 10),(3, 'name3',
30),(5, 'name5', 50),(7, 'name7', 70),(9, 'name9', 90),(10, 'name10', 100);
9. QUERY again should work fine
select * from test_adpt_int where id = 6;
select * from test_adpt_int where id = 5;
```
besides, the bloom index folder looks like below:
The segment generated in 1.4.1 do not have version info file while the
segment generated in 1.5.0 has the version info.
*Note:* in 1.5.0, we introduce the 'mergeShard' to merge the bloom index
file.
---
[GitHub] carbondata issue #2751: [CARBONDATA-2946] Add bloomindex version info file f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2751 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/460/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2749 @ravipesala: PR is ready. Please check ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2738 Thanks. ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2738 ok, I will try to support C to write carbon file later. ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2738 @xubo245 Yes, another team of our department use C, now they use C to write parquet file. ---
[GitHub] carbondata issue #2756: [CARBONDATA-2966]Update Documentation For Avro DataT...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2756 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/639/ ---
[GitHub] carbondata issue #2756: [CARBONDATA-2966]Update Documentation For Avro DataT...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2756 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8709/ ---
[GitHub] carbondata pull request #2736: [CARBONDATA-2947] Adaptive encoding support f...
Github user kunal642 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2736#discussion_r219947411
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/executor/util/QueryUtil.java
---
@@ -781,4 +781,34 @@ public static void putDataToVector(CarbonColumnVector
vector, byte[] value, int
}
}
}
+
+ /**
+ * Put the data to vector
+ *
+ * @param vector
+ * @param value
+ * @param vectorRow
+ */
+ public static void putDataToVector(CarbonColumnVector vector, Object
value, int vectorRow) {
+DataType dt = vector.getType();
+if (value.equals(CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY) ||
value
+.equals(CarbonCommonConstants.EMPTY_BYTE_ARRAY)) {
+ vector.putNull(vectorRow);
+} else {
+ if (dt == DataTypes.STRING) {
+vector.putBytes(vectorRow, (byte[]) value);
+ } else if (dt == DataTypes.BOOLEAN) {
+vector.putBoolean(vectorRow, (boolean) value);
+ } else if (dt == DataTypes.SHORT) {
+vector.putShort(vectorRow, (short) value);
+ } else if (dt == DataTypes.INT) {
+vector.putInt(vectorRow, (int) value);
+ } else if (dt == DataTypes.LONG) {
--- End diff --
Please handle for byte also
---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8708/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8707/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/638/ ---
[GitHub] carbondata issue #2756: [CARBONDATA-2966]Update Documentation For Avro DataT...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2756 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/459/ ---
[GitHub] carbondata pull request #2756: [CARBONDATA-2966]Update Documentation For Avr...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2756 [CARBONDATA-2966]Update Documentation For Avro DataType conversion Updated document for Avro DataType conversion to carbon - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata doc_avro Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2756.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2756 commit f30d1489dbdbe3fecd9eacb96ddff3658904b691 Author: Indhumathi27 Date: 2018-09-24T18:04:04Z [CARBONDATA-2966]Update Documentation For Avro DataType conversion ---
[GitHub] carbondata issue #2751: [CARBONDATA-2946] Add bloomindex version info file f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2751 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8706/ ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8705/ ---
[jira] [Created] (CARBONDATA-2966) Update Documentation For Avro DataType conversion
Indhumathi Muthumurugesh created CARBONDATA-2966: Summary: Update Documentation For Avro DataType conversion Key: CARBONDATA-2966 URL: https://issues.apache.org/jira/browse/CARBONDATA-2966 Project: CarbonData Issue Type: Improvement Reporter: Indhumathi Muthumurugesh Assignee: Indhumathi Muthumurugesh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/637/ ---
[GitHub] carbondata issue #2751: [CARBONDATA-2946] Add bloomindex version info file f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2751 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/636/ ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/635/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2665 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/458/ ---
[GitHub] carbondata issue #2730: [WIP] Performance improvements for Fileformat and Pr...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2730 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/457/ ---
[GitHub] carbondata issue #2751: [CARBONDATA-2946] Add bloomindex version info file f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2751 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/456/ ---
[GitHub] carbondata issue #2754: [WIP] Fixed data loading failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2754 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/634/ ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2738 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8703/ ---
[GitHub] carbondata issue #2754: [WIP] Fixed data loading failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2754 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8702/ ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8704/ ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/455/ ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/633/ ---
[GitHub] carbondata issue #2754: [WIP] Fixed data loading failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2754 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/454/ ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2738 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/632/ ---
[GitHub] carbondata issue #2755: [CARBONDATA-2965] Support scan performance benchmark...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2755 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/631/ ---
[GitHub] carbondata issue #2755: [CARBONDATA-2965] Support scan performance benchmark...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2755 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8701/ ---
[GitHub] carbondata issue #2665: [CARBONDATA-2897][DataMap] Optimize datamap chooser
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2665 retest this please ---
[GitHub] carbondata pull request #2752: [WIP]Even after carbon file is copied to targ...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2752#discussion_r219894302
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -266,16 +268,18 @@ private void notifyDataMapBlockEnd() {
* Finish writing current file. It will flush stream, copy and rename
temp file to final file
* @param copyInCurrentThread set to false if want to do data copy in a
new thread
*/
- protected void commitCurrentFile(boolean copyInCurrentThread) {
+ protected void commitCurrentFile(boolean copyInCurrentThread) throws
IOException {
notifyDataMapBlockEnd();
CarbonUtil.closeStreams(this.fileOutputStream, this.fileChannel);
-if (!enableDirectlyWriteData2Hdfs) {
- if (copyInCurrentThread) {
-
CarbonUtil.copyCarbonDataFileToCarbonStorePath(carbonDataFileTempPath,
-model.getCarbonDataDirectoryPath(), fileSizeInBytes);
- } else {
-executorServiceSubmitList.add(executorService.submit(
-new CompleteHdfsBackendThread(carbonDataFileTempPath)));
+if (null != carbonDataFileTempPath &&
FileFactory.isFileExist(carbonDataFileTempPath)) {
--- End diff --
Why do we need this judgement? In what circumstance will the temp file do
not exist?
---
[GitHub] carbondata pull request #2752: [WIP]Even after carbon file is copied to targ...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2752#discussion_r219893862
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -418,6 +423,24 @@ protected void closeExecutorService() throws
CarbonDataWriterException {
CarbonDataWriterException exception = null;
try {
listener.finish();
+ String tableName = this.model.getTableName();
+ // Delete CarbonData and CarbonIndex files after copying from temp
location
+ if (null != carbonDataFileTempPath && null ==
model.getTableSpec().getCarbonTable()
+ .getBucketingInfo(tableName)) {
+String deleteDataAndIndexInTempPath =
--- End diff --
Why not delete this path recursively so that we do not need the traverse
and delete them one by one?
You can find the delete* methods in `CarbonUtil`
---
[GitHub] carbondata issue #2751: [CARBONDATA-2946] Add bloomindex version info file f...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2751 retest this please ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/453/ ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2738 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/452/ ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8698/ ---
[GitHub] carbondata issue #2755: [CARBONDATA-2965] Support scan performance benchmark...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2755 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/451/ ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/2736 retest this please ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/628/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/627/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8697/ ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8696/ ---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2738 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/450/ ---
[GitHub] carbondata pull request #2755: [CARBONDATA-2965] Support scan performance be...
GitHub user jackylk opened a pull request: https://github.com/apache/carbondata/pull/2755 [CARBONDATA-2965] Support scan performance benchmark in CarbonCli tool Scan performance benchmarking is added as a command in CarbonCli tool. Example usage: shell>java -jar carbondata-cli.jar org.apache.carbondata.CarbonCli -cmd benchmark -p hdfs://carbon1:9000/carbon.store/tpchcarbon_base/lineitem/ -a -c l_comment outputs ``` ## Benchmark ReadHeaderAndFooter takes 12,598 us ConvertFooter takes 4,712 us ReadAllMetaAndConvertFooter takes 8,039 us Scan column 'l_comment' Blocklet#0: ColumnChunkIO takes 222,609 us Blocklet#0: DecompressPage takes 111,985 us Blocklet#1: ColumnChunkIO takes 186,522 us Blocklet#1: DecompressPage takes 89,132 us Blocklet#2: ColumnChunkIO takes 209,129 us Blocklet#2: DecompressPage takes 84,051 us ``` - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/incubator-carbondata scan-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2755.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2755 commit b8dce3a935c9ea1df45ccc4135834cb2f21c06cf Author: Jacky Li Date: 2018-09-22T16:01:04Z support Benchmark command in CarbonCli commit 9fed41fcfb9b1f2900029db5ab0a3a8610b4085e Author: Jacky Li Date: 2018-09-24T15:10:54Z fix test ---
[jira] [Created] (CARBONDATA-2965) Support scan performance benchmark tool
Jacky Li created CARBONDATA-2965: Summary: Support scan performance benchmark tool Key: CARBONDATA-2965 URL: https://issues.apache.org/jira/browse/CARBONDATA-2965 Project: CarbonData Issue Type: New Feature Reporter: Jacky Li Assignee: Jacky Li -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2738: [CARBONDATA-2952] Provide c++ interface for S...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2738#discussion_r219873751
--- Diff: store/CSDK/main.cpp ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include "CarbonReader.h"
+
+using namespace std;
+
+JavaVM *jvm;
+
+JNIEnv *initJVM() {
+JNIEnv *env;
+JavaVMInitArgs vm_args;
+int parNum = 3;
+int res;
+JavaVMOption options[parNum];
+
+options[0].optionString = "-Djava.compiler=NONE";
+options[1].optionString =
"-Djava.class.path=/Users/xubo/Desktop/xubo/git/c/test/jar/carbondata-sdk.jar";
--- End diff --
optimized
---
[GitHub] carbondata issue #2738: [CARBONDATA-2952] Provide c++ interface for SDK Carb...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2738 @zzcclp Now we plan support C++, do you need C interface of carbon? ---
[GitHub] carbondata issue #2754: [WIP] Fixed data loading failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2754 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/449/ ---
[GitHub] carbondata issue #2747: [CARBONDATA-2960] SDK Reader fix with projection col...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2747 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/625/ ---
[GitHub] carbondata issue #2754: [WIP] Fixed data loading failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2754 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/629/ ---
[GitHub] carbondata issue #2754: [WIP] Fixed data loading failure
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2754 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8699/ ---
[GitHub] carbondata pull request #2754: [HOTFIX] Fixed data loading failure
GitHub user kumarvishal09 opened a pull request: https://github.com/apache/carbondata/pull/2754 [HOTFIX] Fixed data loading failure Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kumarvishal09/incubator-carbondata dataloadfailure Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2754.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2754 commit f0e07727d29800a034c43b50b844d47a54d0d80f Author: kumarvishal09 Date: 2018-09-24T14:04:59Z fixed issue ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/448/ ---
[GitHub] carbondata issue #2747: [CARBONDATA-2960] SDK Reader fix with projection col...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2747 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8695/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/447/ ---
[GitHub] carbondata issue #2753: [CARBONDATA-2964] Fix for unsupported float data typ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2753 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8694/ ---
[GitHub] carbondata issue #2753: [CARBONDATA-2964] Fix for unsupported float data typ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2753 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/624/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2749 LGTM ---
[GitHub] carbondata issue #2753: [CARBONDATA-2964] Fix for unsupported float data typ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2753 LGTM ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2736 LGTM ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8693/ ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/446/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/623/ ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user dhatchayani commented on the issue: https://github.com/apache/carbondata/pull/2736 retest this please ---
[GitHub] carbondata issue #2747: [CARBONDATA-2960] SDK Reader fix with projection col...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2747 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/445/ ---
[GitHub] carbondata issue #2747: [CARBONDATA-2960] SDK Reader fix with projection col...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2747 add to whitelist ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/618/ ---
[GitHub] carbondata issue #2753: [CARBONDATA-2964] Fix for unsupported float data typ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2753 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/444/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/443/ ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8688/ ---
[GitHub] carbondata issue #2747: [CARBONDATA-2960] SDK Reader fix with projection col...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2747 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/442/ ---
[GitHub] carbondata issue #2747: [CARBONDATA-2960] SDK Reader fix with projection col...
Github user kumarvishal09 commented on the issue: https://github.com/apache/carbondata/pull/2747 retest this please ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8689/ ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/619/ ---
[jira] [Resolved] (CARBONDATA-2956) CarbonReader can't support use configuration to read S3 data
[
https://issues.apache.org/jira/browse/CARBONDATA-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Kapoor resolved CARBONDATA-2956.
--
Resolution: Fixed
> CarbonReader can't support use configuration to read S3 data
>
>
> Key: CARBONDATA-2956
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2956
> Project: CarbonData
> Issue Type: Bug
> Components: other
>Affects Versions: 1.5.0
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
> Fix For: 1.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Test code:
> {code:java}
> package org.apache.carbondata.examples.sdk;
> import org.apache.carbondata.common.logging.LogService;
> import org.apache.carbondata.common.logging.LogServiceFactory;
> import org.apache.carbondata.core.metadata.datatype.DataTypes;
> import org.apache.carbondata.core.scan.expression.ColumnExpression;
> import org.apache.carbondata.core.scan.expression.LiteralExpression;
> import
> org.apache.carbondata.core.scan.expression.conditional.EqualToExpression;
> import org.apache.carbondata.sdk.file.*;
> import org.apache.hadoop.conf.Configuration;
> import static org.apache.hadoop.fs.s3a.Constants.ACCESS_KEY;
> import static org.apache.hadoop.fs.s3a.Constants.ENDPOINT;
> import static org.apache.hadoop.fs.s3a.Constants.SECRET_KEY;
> /**
> * Example for testing CarbonWriter on S3
> */
> public class SDKS3Example {
> public static void main(String[] args) throws Exception {
> LogService logger =
> LogServiceFactory.getLogService(SDKS3Example.class.getName());
> if (args == null || args.length < 3) {
> logger.error("Usage: java CarbonS3Example:
> "
> + " [table-path-on-s3] [rows]");
> System.exit(0);
> }
> String path = "s3a://sdk/WriterOutput";
> if (args.length > 3) {
> path=args[3];
> }
> int num = 3;
> if (args.length > 4) {
> num = Integer.parseInt(args[4]);
> }
> Configuration conf = new Configuration();
> conf.set(ACCESS_KEY,args[0]);
> conf.set(SECRET_KEY,args[1]);
> conf.set(ENDPOINT,args[2]);
> //Field[] fields = new Field[2];
> //fields[0] = new Field("name", DataTypes.STRING);
> //fields[1] = new Field("age", DataTypes.INT);
> //CarbonWriterBuilder builder = CarbonWriter.builder()
> //.setAccessKey(args[0])
> //.setSecretKey(args[1])
> //.setEndPoint(args[2])
> //.outputPath(path);
> //
> //CarbonWriter writer = builder.buildWriterForCSVInput(new
> Schema(fields), conf);
> //
> //for (int i = 0; i < num; i++) {
> //writer.write(new String[]{"robot" + (i % 10),
> String.valueOf(i)});
> //}
> //writer.close();
> // Read data
> EqualToExpression equalToExpression = new EqualToExpression(
> new ColumnExpression("name", DataTypes.STRING),
> new LiteralExpression("robot1", DataTypes.STRING));
> CarbonReader reader = CarbonReader
> .builder(path, "_temp")
> .projection(new String[]{"name", "age"})
> .filter(equalToExpression)
> .build(conf);
> System.out.println("\nData:");
> int i = 0;
> while (i < 20 && reader.hasNext()) {
> Object[] row = (Object[]) reader.readNextRow();
> System.out.println(row[0] + " " + row[1]);
> i++;
> }
> System.out.println("\nFinished");
> reader.close();
> }
> }
> {code}
> Exception:
> {code:java}
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.util.Shell).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more
> info.
> Exception in thread "main" com.amazonaws.AmazonClientException: Unable to
> load AWS credentials from any provider in the chain
> at
> com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3521)
> at
> com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
> at
> com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at
[GitHub] carbondata pull request #2742: [CARBONDATA-2956] CarbonReader support use co...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2742 ---
[GitHub] carbondata issue #2742: [CARBONDATA-2956] CarbonReader support use configura...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2742 LGTM ---
[GitHub] carbondata issue #2736: [CARBONDATA-2947] Adaptive encoding support for time...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2736 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/440/ ---
[GitHub] carbondata issue #2752: [WIP]Even after carbon file is copied to targetfolde...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2752 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/439/ ---
[jira] [Updated] (CARBONDATA-2947) Adaptive encoding support for timestamp no dictionary and Refactor ColumnPageWrapper
[ https://issues.apache.org/jira/browse/CARBONDATA-2947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhatchayani updated CARBONDATA-2947: Summary: Adaptive encoding support for timestamp no dictionary and Refactor ColumnPageWrapper (was: Adaptive encoding support for timestamp no dictionary) > Adaptive encoding support for timestamp no dictionary and Refactor > ColumnPageWrapper > > > Key: CARBONDATA-2947 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2947 > Project: CarbonData > Issue Type: Improvement >Reporter: dhatchayani >Assignee: dhatchayani >Priority: Minor > Time Spent: 6h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2742: [CARBONDATA-2956] CarbonReader support use configura...
Github user kunal642 commented on the issue: https://github.com/apache/carbondata/pull/2742 LGTM ---
[GitHub] carbondata pull request #2746: [Compatibility] fix legacy store compatibilit...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2746 ---
[GitHub] carbondata issue #2753: [CARBONDATA-2964] Fix for unsupported float data typ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2753 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/438/ ---
[GitHub] carbondata issue #2746: [Compatibility] fix legacy store compatibility issue
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2746 LGTM ---
[jira] [Resolved] (CARBONDATA-2963) Add support to add byte column as a sort column
[ https://issues.apache.org/jira/browse/CARBONDATA-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravindra Pesala resolved CARBONDATA-2963. - Resolution: Fixed Fix Version/s: 1.5.0 > Add support to add byte column as a sort column > --- > > Key: CARBONDATA-2963 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2963 > Project: CarbonData > Issue Type: Improvement >Reporter: Kunal Kapoor >Priority: Minor > Fix For: 1.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2750: [CARBONDATA-2963] added support for byte as s...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2750 ---
[GitHub] carbondata issue #2747: [CARBONDATA-2960] SDK Reader fix with projection col...
Github user shardul-cr7 commented on the issue: https://github.com/apache/carbondata/pull/2747 retest this please ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8686/ ---
[GitHub] carbondata issue #2746: [Compatibility] fix legacy store compatibility issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2746 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/615/ ---
[GitHub] carbondata issue #2746: [Compatibility] fix legacy store compatibility issue
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2746 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8685/ ---
[GitHub] carbondata issue #2749: [CARBONDATA-2961] Simplify SDK API interfaces
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2749 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/616/ ---
