[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1082/ ---
[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9149/ ---
[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/884/ ---
[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2792 retest this please ---
[GitHub] carbondata pull request #2819: [CARBONDATA-3012] Added support for full scan...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2819#discussion_r226832821
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/datatype/DecimalConverterFactory.java
---
@@ -173,6 +221,23 @@ public int getSizeInBytes() {
return new BigDecimal(bigInteger, scale);
}
+@Override public void fillVector(Object valuesToBeConverted, int size,
ColumnVectorInfo info,
+BitSet nullBitset) {
+ CarbonColumnVector vector = info.vector;
+ int precision = info.measure.getMeasure().getPrecision();
+ if (valuesToBeConverted instanceof byte[][]) {
+byte[][] data = (byte[][]) valuesToBeConverted;
+for (int i = 0; i < size; i++) {
+ if (nullBitset.get(i)) {
+vector.putNull(i);
+ } else {
+BigInteger bigInteger = new BigInteger(data[i]);
+vector.putDecimal(i, new BigDecimal(bigInteger, scale),
precision);
--- End diff --
The method `DataTypeUtil.byteToBigDecimal(data[i]), precision)` is
different as it calculates scale also from binary, but here we no need of it.
---
[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1081/ ---
[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2792 Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9148/ ---
[GitHub] carbondata issue #2792: [CARBONDATA-2981] Support read primitive data type i...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2792 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/883/ ---
[jira] [Updated] (CARBONDATA-3025) Add SQL support for cli, and enhance CLI , add more metadata to carbon file
[ https://issues.apache.org/jira/browse/CARBONDATA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akash R Nilugal updated CARBONDATA-3025: Attachment: Carbon_Maintainability.odt > Add SQL support for cli, and enhance CLI , add more metadata to carbon file > --- > > Key: CARBONDATA-3025 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3025 > Project: CarbonData > Issue Type: Bug >Reporter: Akash R Nilugal >Assignee: Akash R Nilugal >Priority: Major > Attachments: Carbon_Maintainability.odt > > Time Spent: 6h 50m > Remaining Estimate: 0h > > # support SQL integration for CLI > # enhancle cli with more info > # add more metadata to carbon file footer for better maintainability -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2792: [CARBONDATA-2981] Support read primitive data...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226830397
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/RowUtil.java ---
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.file;
+
+import java.io.Serializable;
+import java.math.BigDecimal;
+
+/**
+ * This row class is used to transfer the row data from one step to other
step
+ */
+public class RowUtil implements Serializable {
+
+ public static String getString(Object[] data, int ordinal) {
+return (String) data[ordinal];
+ }
+
+ /**
+ * get short type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return short data type data
+ */
+ public static short getShort(Object[] data, int ordinal) {
+return (short) data[ordinal];
+ }
+
+ /**
+ * get int data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return int data type data
+ */
+ public static int getInt(Object[] data, int ordinal) {
+return (Integer) data[ordinal];
+ }
+
+ /**
+ * get long data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return long data type data
+ */
+ public static long getLong(Object[] data, int ordinal) {
+return (long) data[ordinal];
+ }
+
+ /**
+ * get array data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return array data type data
+ */
+ public static Object[] getArray(Object[] data, int ordinal) {
+return (Object[]) data[ordinal];
+ }
+
+ /**
+ * get double data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return double data type data
+ */
+ public static double getDouble(Object[] data, int ordinal) {
+return (double) data[ordinal];
+ }
+
+ /**
+ * get boolean data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return boolean data type data
+ */
+ public static boolean getBoolean(Object[] data, int ordinal) {
+return (boolean) data[ordinal];
+ }
+
+ /**
+ * get byte data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return byte data type data
+ */
+ public static Byte getByte(Object[] data, int ordinal) {
+return (Byte) data[ordinal];
+ }
+
+ /**
+ * get float data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return float data type data
+ */
+ public static float getFloat(Object[] data, int ordinal) {
+return (float) data[ordinal];
+ }
+
+ /**
+ * get varchar data type data by ordinal
+ * This is for CSDK
+ * JNI don't support varchar, so carbon convert decimal to string
--- End diff --
sorry, I changed, please check.
---
[GitHub] carbondata pull request #2822: [CARBONDATA-3014] Added support for inverted ...
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2822#discussion_r226830306
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/scanner/impl/BlockletFilterScanner.java
---
@@ -316,4 +320,167 @@ private BlockletScannedResult
executeFilter(RawBlockletColumnChunks rawBlockletC
readTime.getCount() + dimensionReadTime);
return scannedResult;
}
+
+ /**
+ * This method will process the data in below order
+ * 1. first apply min max on the filter tree and check whether any of
the filter
+ * is fall on the range of min max, if not then return empty result
+ * 2. If filter falls on min max range then apply filter on actual
+ * data and get the pruned pages.
+ * 3. if pruned pages are not empty then read only those blocks(measure
or dimension)
+ * which was present in the query but not present in the filter, as
while applying filter
+ * some of the blocks where already read and present in chunk holder so
not need to
+ * read those blocks again, this is to avoid reading of same blocks
which was already read
+ * 4. Set the blocks and filter pages to scanned result
+ *
+ * @param rawBlockletColumnChunks blocklet raw chunk of all columns
+ * @throws FilterUnsupportedException
+ */
+ private BlockletScannedResult executeFilterForPages(
+ RawBlockletColumnChunks rawBlockletColumnChunks)
+ throws FilterUnsupportedException, IOException {
+long startTime = System.currentTimeMillis();
+QueryStatistic totalBlockletStatistic =
queryStatisticsModel.getStatisticsTypeAndObjMap()
+.get(QueryStatisticsConstants.TOTAL_BLOCKLET_NUM);
+
totalBlockletStatistic.addCountStatistic(QueryStatisticsConstants.TOTAL_BLOCKLET_NUM,
+totalBlockletStatistic.getCount() + 1);
+// apply filter on actual data, for each page
+BitSet pages = this.filterExecuter.prunePages(rawBlockletColumnChunks);
+// if filter result is empty then return with empty result
+if (pages.isEmpty()) {
+
CarbonUtil.freeMemory(rawBlockletColumnChunks.getDimensionRawColumnChunks(),
+ rawBlockletColumnChunks.getMeasureRawColumnChunks());
+
+ QueryStatistic scanTime =
queryStatisticsModel.getStatisticsTypeAndObjMap()
+ .get(QueryStatisticsConstants.SCAN_BLOCKlET_TIME);
+
scanTime.addCountStatistic(QueryStatisticsConstants.SCAN_BLOCKlET_TIME,
+ scanTime.getCount() + (System.currentTimeMillis() - startTime));
+
+ QueryStatistic scannedPages =
queryStatisticsModel.getStatisticsTypeAndObjMap()
+ .get(QueryStatisticsConstants.PAGE_SCANNED);
+ scannedPages.addCountStatistic(QueryStatisticsConstants.PAGE_SCANNED,
+ scannedPages.getCount());
+ return createEmptyResult();
+}
+
+BlockletScannedResult scannedResult =
+new FilterQueryScannedResult(blockExecutionInfo,
queryStatisticsModel);
+
+// valid scanned blocklet
+QueryStatistic validScannedBlockletStatistic =
queryStatisticsModel.getStatisticsTypeAndObjMap()
+.get(QueryStatisticsConstants.VALID_SCAN_BLOCKLET_NUM);
+validScannedBlockletStatistic
+
.addCountStatistic(QueryStatisticsConstants.VALID_SCAN_BLOCKLET_NUM,
+validScannedBlockletStatistic.getCount() + 1);
+// adding statistics for valid number of pages
+QueryStatistic validPages =
queryStatisticsModel.getStatisticsTypeAndObjMap()
+.get(QueryStatisticsConstants.VALID_PAGE_SCANNED);
+
validPages.addCountStatistic(QueryStatisticsConstants.VALID_PAGE_SCANNED,
+validPages.getCount() + pages.cardinality());
+QueryStatistic scannedPages =
queryStatisticsModel.getStatisticsTypeAndObjMap()
+.get(QueryStatisticsConstants.PAGE_SCANNED);
+scannedPages.addCountStatistic(QueryStatisticsConstants.PAGE_SCANNED,
+scannedPages.getCount() + pages.cardinality());
+// get the row indexes from bit set for each page
+int[] pageFilteredPages = new int[pages.cardinality()];
+int index = 0;
+for (int i = pages.nextSetBit(0); i >= 0; i = pages.nextSetBit(i + 1))
{
+ pageFilteredPages[index++] = i;
+}
+// count(*) case there would not be any dimensions are measures
selected.
+int[] numberOfRows = new int[pages.cardinality()];
+for (int i = 0; i < numberOfRows.length; i++) {
+ numberOfRows[i] =
rawBlockletColumnChunks.getDataBlock().getPageRowCount(i);
--- End diff --
ok
---
[GitHub] carbondata issue #2838: [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT
Github user zzcclp commented on the issue: https://github.com/apache/carbondata/pull/2838 LGTM ---
[GitHub] carbondata pull request #2838: [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2838 ---
[GitHub] carbondata issue #2838: [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2838 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1080/ ---
[GitHub] carbondata issue #2838: [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2838 Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9147/ ---
[GitHub] carbondata issue #2838: [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2838 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/882/ ---
[GitHub] carbondata pull request #2838: [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT
GitHub user chenliang613 opened a pull request: https://github.com/apache/carbondata/pull/2838 [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT Upgrade pom version to 1.6-SNAPSHOT You can merge this pull request into a Git repository by running: $ git pull https://github.com/chenliang613/carbondata upgrade_1.6 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2838.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2838 commit 01fe9f12fbadec8f8fcada8f183ec8c7faa4b6b5 Author: chenliang613 Date: 2018-10-20T09:49:25Z [HOTFIX] Upgrade pom version to 1.6-SNAPSHOT ---
[GitHub] carbondata pull request #2792: [CARBONDATA-2981] Support read primitive data...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2792#discussion_r226821049
--- Diff:
store/sdk/src/main/java/org/apache/carbondata/sdk/file/RowUtil.java ---
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.file;
+
+import java.io.Serializable;
+import java.math.BigDecimal;
+
+/**
+ * This row class is used to transfer the row data from one step to other
step
+ */
+public class RowUtil implements Serializable {
+
+ public static String getString(Object[] data, int ordinal) {
+return (String) data[ordinal];
+ }
+
+ /**
+ * get short type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return short data type data
+ */
+ public static short getShort(Object[] data, int ordinal) {
+return (short) data[ordinal];
+ }
+
+ /**
+ * get int data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return int data type data
+ */
+ public static int getInt(Object[] data, int ordinal) {
+return (Integer) data[ordinal];
+ }
+
+ /**
+ * get long data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return long data type data
+ */
+ public static long getLong(Object[] data, int ordinal) {
+return (long) data[ordinal];
+ }
+
+ /**
+ * get array data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return array data type data
+ */
+ public static Object[] getArray(Object[] data, int ordinal) {
+return (Object[]) data[ordinal];
+ }
+
+ /**
+ * get double data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return double data type data
+ */
+ public static double getDouble(Object[] data, int ordinal) {
+return (double) data[ordinal];
+ }
+
+ /**
+ * get boolean data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return boolean data type data
+ */
+ public static boolean getBoolean(Object[] data, int ordinal) {
+return (boolean) data[ordinal];
+ }
+
+ /**
+ * get byte data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return byte data type data
+ */
+ public static Byte getByte(Object[] data, int ordinal) {
+return (Byte) data[ordinal];
+ }
+
+ /**
+ * get float data type data by ordinal
+ *
+ * @param data carbon row data
+ * @param ordinal the data index of Row
+ * @return float data type data
+ */
+ public static float getFloat(Object[] data, int ordinal) {
+return (float) data[ordinal];
+ }
+
+ /**
+ * get varchar data type data by ordinal
+ * This is for CSDK
+ * JNI don't support varchar, so carbon convert decimal to string
--- End diff --
but this method is for varchar not decimal...
---
