[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine
CarbonDataQA1 commented on pull request #3885: URL: https://github.com/apache/carbondata/pull/3885#issuecomment-673905745 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1976/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3885: [CARBONDATA-3946] Support IndexServer with Presto Engine
CarbonDataQA1 commented on pull request #3885: URL: https://github.com/apache/carbondata/pull/3885#issuecomment-673905024 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3715/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3778: [CARBONDATA-3916] Support array complex type with SI
CarbonDataQA1 commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-673883264 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3716/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3778: [CARBONDATA-3916] Support array complex type with SI
CarbonDataQA1 commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-673882872 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1977/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3778: [CARBONDATA-3916] Support array complex type with SI
CarbonDataQA1 commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-673649958 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1974/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3778: [CARBONDATA-3916] Support array complex type with SI
CarbonDataQA1 commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-673637872 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3713/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3890: [WIP] After reset query not hitting MV
CarbonDataQA1 commented on pull request #3890: URL: https://github.com/apache/carbondata/pull/3890#issuecomment-673590822 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3712/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3890: [WIP] After reset query not hitting MV
CarbonDataQA1 commented on pull request #3890: URL: https://github.com/apache/carbondata/pull/3890#issuecomment-673590190 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1973/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3887: [WIP] Refactor #3773 and support struct type
CarbonDataQA1 commented on pull request #3887: URL: https://github.com/apache/carbondata/pull/3887#issuecomment-673588956 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3714/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3887: [WIP] Refactor #3773 and support struct type
CarbonDataQA1 commented on pull request #3887: URL: https://github.com/apache/carbondata/pull/3887#issuecomment-673588446 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1975/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3837: [CARBONDATA-3927]Remove unwanted fields from tupleID to make it short and to improve store size and performance.
CarbonDataQA1 commented on pull request #3837: URL: https://github.com/apache/carbondata/pull/3837#issuecomment-673586193 Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1972/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3837: [CARBONDATA-3927]Remove unwanted fields from tupleID to make it short and to improve store size and performance.
CarbonDataQA1 commented on pull request #3837: URL: https://github.com/apache/carbondata/pull/3837#issuecomment-673577314 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3711/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG opened a new pull request #3890: [WIP] After reset query not hitting MV
ShreelekhyaG opened a new pull request #3890: URL: https://github.com/apache/carbondata/pull/3890 ### Why is this PR needed? After reset query not hitting MV. With the `reset`, `spark.sql.warehouse.dir` and `carbonStorePath` don't match and the databaseLocation will change to old table path format and refers to wrong mv_status file. ### What changes were proposed in this PR? Made changes to get the mv status file path from `SchemaProvider` so it can return the path if it is already stored in the map. ### Does this PR introduce any user interface change? - No ### Is any new testcase added? - Yes This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3855: [CARBONDATA-3863], after using index service clean the temp data
CarbonDataQA1 commented on pull request #3855: URL: https://github.com/apache/carbondata/pull/3855#issuecomment-673478731 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1971/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3855: [CARBONDATA-3863], after using index service clean the temp data
CarbonDataQA1 commented on pull request #3855: URL: https://github.com/apache/carbondata/pull/3855#issuecomment-673462582 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3710/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI
CarbonDataQA1 commented on pull request #3876: URL: https://github.com/apache/carbondata/pull/3876#issuecomment-673455327 Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3709/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI
CarbonDataQA1 commented on pull request #3876: URL: https://github.com/apache/carbondata/pull/3876#issuecomment-673453726 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1970/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] MarvinLitt commented on a change in pull request #3855: [CARBONDATA-3863], after using index service clean the temp data
MarvinLitt commented on a change in pull request #3855: URL: https://github.com/apache/carbondata/pull/3855#discussion_r469870300 ## File path: integration/spark/src/test/scala/org/apache/indexserver/IndexServerTest.scala ## @@ -0,0 +1,56 @@ +package org.apache.indexserver Review comment: done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3773: [CARBONDATA-3830]Presto array columns read support
ajantha-bhat commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r469863077
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/FillVector.java
##
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.ColumnVectorInfo;
+import
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+import org.apache.carbondata.core.util.ByteUtil;
+
+public class FillVector {
+ private byte[] pageData;
+ private float floatFactor = 0;
+ private double factor = 0;
+ private ColumnVectorInfo vectorInfo;
+ private BitSet nullBits;
+
+ public FillVector(byte[] pageData, ColumnVectorInfo vectorInfo, BitSet
nullBits) {
+this.pageData = pageData;
+this.vectorInfo = vectorInfo;
+this.nullBits = nullBits;
+ }
+
+ public void setFactor(double factor) {
+this.factor = factor;
+ }
+
+ public void setFloatFactor(float floatFactor) {
+this.floatFactor = floatFactor;
+ }
+
+ public void basedOnType(CarbonColumnVector vector, DataType vectorDataType,
int pageSize,
+ DataType pageDataType) {
+if (vectorInfo.vector.getColumnVector() != null &&
((CarbonColumnVectorImpl) vectorInfo.vector
+.getColumnVector()).isComplex()) {
+ fillComplexType(vector.getColumnVector(), pageDataType);
+} else {
+ fillPrimitiveType(vector, vectorDataType, pageSize, pageDataType);
+ vector.setIndex(0);
+}
+ }
+
+ private void fillComplexType(CarbonColumnVector vector, DataType
pageDataType) {
+CarbonColumnVectorImpl vectorImpl = (CarbonColumnVectorImpl) vector;
+if (vector != null && vector.getChildrenVector() != null) {
+ ArrayList childElements = ((CarbonColumnVectorImpl)
vector).getChildrenElements();
+ for (int i = 0; i < childElements.size(); i++) {
+int count = childElements.get(i);
+typeComplexObject(vectorImpl.getChildrenVector().get(0), count,
pageDataType);
+vector.putArrayObject();
+ }
+ vectorImpl.getChildrenVector().get(0).setIndex(0);
+}
+ }
+
+ private void fillPrimitiveType(CarbonColumnVector vector, DataType
vectorDataType, int pageSize,
+ DataType pageDataType) {
+// offset which denotes the start index for pageData
+int pageIndex = vector.getIndex();
+int rowId = 0;
+
+// Filling into vector is done based on page data type
+
+if (vectorDataType == DataTypes.FLOAT && floatFactor != 0.0) {
+ if (pageDataType == DataTypes.BOOLEAN || pageDataType == DataTypes.BYTE)
{
+for (int i = 0; i < pageSize; i++) {
+ vector.putFloat(i, (pageData[pageIndex++] / floatFactor));
+}
+ } else if (pageDataType == DataTypes.SHORT) {
+int size = pageSize * DataTypes.SHORT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toShortLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.SHORT_INT) {
+int size = pageSize * DataTypes.SHORT_INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT_INT.getSizeInBytes()) {
+ vector.putFloat(rowId++, (ByteUtil.valueOf3Bytes(pageData, pageIndex
+ i) / floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.INT) {
+int size = pageSize * DataTypes.INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.INT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toIntLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageInde
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3773: [CARBONDATA-3830]Presto array columns read support
ajantha-bhat commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r469858716
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/FillVector.java
##
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.ColumnVectorInfo;
+import
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+import org.apache.carbondata.core.util.ByteUtil;
+
+public class FillVector {
+ private byte[] pageData;
+ private float floatFactor = 0;
+ private double factor = 0;
+ private ColumnVectorInfo vectorInfo;
+ private BitSet nullBits;
+
+ public FillVector(byte[] pageData, ColumnVectorInfo vectorInfo, BitSet
nullBits) {
+this.pageData = pageData;
+this.vectorInfo = vectorInfo;
+this.nullBits = nullBits;
+ }
+
+ public void setFactor(double factor) {
+this.factor = factor;
+ }
+
+ public void setFloatFactor(float floatFactor) {
+this.floatFactor = floatFactor;
+ }
+
+ public void basedOnType(CarbonColumnVector vector, DataType vectorDataType,
int pageSize,
+ DataType pageDataType) {
+if (vectorInfo.vector.getColumnVector() != null &&
((CarbonColumnVectorImpl) vectorInfo.vector
+.getColumnVector()).isComplex()) {
+ fillComplexType(vector.getColumnVector(), pageDataType);
+} else {
+ fillPrimitiveType(vector, vectorDataType, pageSize, pageDataType);
+ vector.setIndex(0);
+}
+ }
+
+ private void fillComplexType(CarbonColumnVector vector, DataType
pageDataType) {
+CarbonColumnVectorImpl vectorImpl = (CarbonColumnVectorImpl) vector;
+if (vector != null && vector.getChildrenVector() != null) {
+ ArrayList childElements = ((CarbonColumnVectorImpl)
vector).getChildrenElements();
+ for (int i = 0; i < childElements.size(); i++) {
+int count = childElements.get(i);
+typeComplexObject(vectorImpl.getChildrenVector().get(0), count,
pageDataType);
+vector.putArrayObject();
+ }
+ vectorImpl.getChildrenVector().get(0).setIndex(0);
+}
+ }
+
+ private void fillPrimitiveType(CarbonColumnVector vector, DataType
vectorDataType, int pageSize,
+ DataType pageDataType) {
+// offset which denotes the start index for pageData
+int pageIndex = vector.getIndex();
+int rowId = 0;
+
+// Filling into vector is done based on page data type
+
+if (vectorDataType == DataTypes.FLOAT && floatFactor != 0.0) {
+ if (pageDataType == DataTypes.BOOLEAN || pageDataType == DataTypes.BYTE)
{
+for (int i = 0; i < pageSize; i++) {
+ vector.putFloat(i, (pageData[pageIndex++] / floatFactor));
+}
+ } else if (pageDataType == DataTypes.SHORT) {
+int size = pageSize * DataTypes.SHORT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toShortLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.SHORT_INT) {
+int size = pageSize * DataTypes.SHORT_INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT_INT.getSizeInBytes()) {
+ vector.putFloat(rowId++, (ByteUtil.valueOf3Bytes(pageData, pageIndex
+ i) / floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.INT) {
+int size = pageSize * DataTypes.INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.INT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toIntLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageInde
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3773: [CARBONDATA-3830]Presto array columns read support
ajantha-bhat commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r469858716
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/FillVector.java
##
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.ColumnVectorInfo;
+import
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+import org.apache.carbondata.core.util.ByteUtil;
+
+public class FillVector {
+ private byte[] pageData;
+ private float floatFactor = 0;
+ private double factor = 0;
+ private ColumnVectorInfo vectorInfo;
+ private BitSet nullBits;
+
+ public FillVector(byte[] pageData, ColumnVectorInfo vectorInfo, BitSet
nullBits) {
+this.pageData = pageData;
+this.vectorInfo = vectorInfo;
+this.nullBits = nullBits;
+ }
+
+ public void setFactor(double factor) {
+this.factor = factor;
+ }
+
+ public void setFloatFactor(float floatFactor) {
+this.floatFactor = floatFactor;
+ }
+
+ public void basedOnType(CarbonColumnVector vector, DataType vectorDataType,
int pageSize,
+ DataType pageDataType) {
+if (vectorInfo.vector.getColumnVector() != null &&
((CarbonColumnVectorImpl) vectorInfo.vector
+.getColumnVector()).isComplex()) {
+ fillComplexType(vector.getColumnVector(), pageDataType);
+} else {
+ fillPrimitiveType(vector, vectorDataType, pageSize, pageDataType);
+ vector.setIndex(0);
+}
+ }
+
+ private void fillComplexType(CarbonColumnVector vector, DataType
pageDataType) {
+CarbonColumnVectorImpl vectorImpl = (CarbonColumnVectorImpl) vector;
+if (vector != null && vector.getChildrenVector() != null) {
+ ArrayList childElements = ((CarbonColumnVectorImpl)
vector).getChildrenElements();
+ for (int i = 0; i < childElements.size(); i++) {
+int count = childElements.get(i);
+typeComplexObject(vectorImpl.getChildrenVector().get(0), count,
pageDataType);
+vector.putArrayObject();
+ }
+ vectorImpl.getChildrenVector().get(0).setIndex(0);
+}
+ }
+
+ private void fillPrimitiveType(CarbonColumnVector vector, DataType
vectorDataType, int pageSize,
+ DataType pageDataType) {
+// offset which denotes the start index for pageData
+int pageIndex = vector.getIndex();
+int rowId = 0;
+
+// Filling into vector is done based on page data type
+
+if (vectorDataType == DataTypes.FLOAT && floatFactor != 0.0) {
+ if (pageDataType == DataTypes.BOOLEAN || pageDataType == DataTypes.BYTE)
{
+for (int i = 0; i < pageSize; i++) {
+ vector.putFloat(i, (pageData[pageIndex++] / floatFactor));
+}
+ } else if (pageDataType == DataTypes.SHORT) {
+int size = pageSize * DataTypes.SHORT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toShortLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.SHORT_INT) {
+int size = pageSize * DataTypes.SHORT_INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT_INT.getSizeInBytes()) {
+ vector.putFloat(rowId++, (ByteUtil.valueOf3Bytes(pageData, pageIndex
+ i) / floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.INT) {
+int size = pageSize * DataTypes.INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.INT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toIntLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageInde
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3773: [CARBONDATA-3830]Presto array columns read support
ajantha-bhat commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r469858716
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/FillVector.java
##
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.ColumnVectorInfo;
+import
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+import org.apache.carbondata.core.util.ByteUtil;
+
+public class FillVector {
+ private byte[] pageData;
+ private float floatFactor = 0;
+ private double factor = 0;
+ private ColumnVectorInfo vectorInfo;
+ private BitSet nullBits;
+
+ public FillVector(byte[] pageData, ColumnVectorInfo vectorInfo, BitSet
nullBits) {
+this.pageData = pageData;
+this.vectorInfo = vectorInfo;
+this.nullBits = nullBits;
+ }
+
+ public void setFactor(double factor) {
+this.factor = factor;
+ }
+
+ public void setFloatFactor(float floatFactor) {
+this.floatFactor = floatFactor;
+ }
+
+ public void basedOnType(CarbonColumnVector vector, DataType vectorDataType,
int pageSize,
+ DataType pageDataType) {
+if (vectorInfo.vector.getColumnVector() != null &&
((CarbonColumnVectorImpl) vectorInfo.vector
+.getColumnVector()).isComplex()) {
+ fillComplexType(vector.getColumnVector(), pageDataType);
+} else {
+ fillPrimitiveType(vector, vectorDataType, pageSize, pageDataType);
+ vector.setIndex(0);
+}
+ }
+
+ private void fillComplexType(CarbonColumnVector vector, DataType
pageDataType) {
+CarbonColumnVectorImpl vectorImpl = (CarbonColumnVectorImpl) vector;
+if (vector != null && vector.getChildrenVector() != null) {
+ ArrayList childElements = ((CarbonColumnVectorImpl)
vector).getChildrenElements();
+ for (int i = 0; i < childElements.size(); i++) {
+int count = childElements.get(i);
+typeComplexObject(vectorImpl.getChildrenVector().get(0), count,
pageDataType);
+vector.putArrayObject();
+ }
+ vectorImpl.getChildrenVector().get(0).setIndex(0);
+}
+ }
+
+ private void fillPrimitiveType(CarbonColumnVector vector, DataType
vectorDataType, int pageSize,
+ DataType pageDataType) {
+// offset which denotes the start index for pageData
+int pageIndex = vector.getIndex();
+int rowId = 0;
+
+// Filling into vector is done based on page data type
+
+if (vectorDataType == DataTypes.FLOAT && floatFactor != 0.0) {
+ if (pageDataType == DataTypes.BOOLEAN || pageDataType == DataTypes.BYTE)
{
+for (int i = 0; i < pageSize; i++) {
+ vector.putFloat(i, (pageData[pageIndex++] / floatFactor));
+}
+ } else if (pageDataType == DataTypes.SHORT) {
+int size = pageSize * DataTypes.SHORT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toShortLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.SHORT_INT) {
+int size = pageSize * DataTypes.SHORT_INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT_INT.getSizeInBytes()) {
+ vector.putFloat(rowId++, (ByteUtil.valueOf3Bytes(pageData, pageIndex
+ i) / floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.INT) {
+int size = pageSize * DataTypes.INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.INT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toIntLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageInde
[GitHub] [carbondata] ajantha-bhat commented on a change in pull request #3773: [CARBONDATA-3830]Presto array columns read support
ajantha-bhat commented on a change in pull request #3773:
URL: https://github.com/apache/carbondata/pull/3773#discussion_r469858716
##
File path:
core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/FillVector.java
##
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.DecimalConverterFactory;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import org.apache.carbondata.core.scan.result.vector.ColumnVectorInfo;
+import
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+import org.apache.carbondata.core.util.ByteUtil;
+
+public class FillVector {
+ private byte[] pageData;
+ private float floatFactor = 0;
+ private double factor = 0;
+ private ColumnVectorInfo vectorInfo;
+ private BitSet nullBits;
+
+ public FillVector(byte[] pageData, ColumnVectorInfo vectorInfo, BitSet
nullBits) {
+this.pageData = pageData;
+this.vectorInfo = vectorInfo;
+this.nullBits = nullBits;
+ }
+
+ public void setFactor(double factor) {
+this.factor = factor;
+ }
+
+ public void setFloatFactor(float floatFactor) {
+this.floatFactor = floatFactor;
+ }
+
+ public void basedOnType(CarbonColumnVector vector, DataType vectorDataType,
int pageSize,
+ DataType pageDataType) {
+if (vectorInfo.vector.getColumnVector() != null &&
((CarbonColumnVectorImpl) vectorInfo.vector
+.getColumnVector()).isComplex()) {
+ fillComplexType(vector.getColumnVector(), pageDataType);
+} else {
+ fillPrimitiveType(vector, vectorDataType, pageSize, pageDataType);
+ vector.setIndex(0);
+}
+ }
+
+ private void fillComplexType(CarbonColumnVector vector, DataType
pageDataType) {
+CarbonColumnVectorImpl vectorImpl = (CarbonColumnVectorImpl) vector;
+if (vector != null && vector.getChildrenVector() != null) {
+ ArrayList childElements = ((CarbonColumnVectorImpl)
vector).getChildrenElements();
+ for (int i = 0; i < childElements.size(); i++) {
+int count = childElements.get(i);
+typeComplexObject(vectorImpl.getChildrenVector().get(0), count,
pageDataType);
+vector.putArrayObject();
+ }
+ vectorImpl.getChildrenVector().get(0).setIndex(0);
+}
+ }
+
+ private void fillPrimitiveType(CarbonColumnVector vector, DataType
vectorDataType, int pageSize,
+ DataType pageDataType) {
+// offset which denotes the start index for pageData
+int pageIndex = vector.getIndex();
+int rowId = 0;
+
+// Filling into vector is done based on page data type
+
+if (vectorDataType == DataTypes.FLOAT && floatFactor != 0.0) {
+ if (pageDataType == DataTypes.BOOLEAN || pageDataType == DataTypes.BYTE)
{
+for (int i = 0; i < pageSize; i++) {
+ vector.putFloat(i, (pageData[pageIndex++] / floatFactor));
+}
+ } else if (pageDataType == DataTypes.SHORT) {
+int size = pageSize * DataTypes.SHORT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toShortLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.SHORT_INT) {
+int size = pageSize * DataTypes.SHORT_INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.SHORT_INT.getSizeInBytes()) {
+ vector.putFloat(rowId++, (ByteUtil.valueOf3Bytes(pageData, pageIndex
+ i) / floatFactor));
+}
+pageIndex += size;
+ } else if (pageDataType == DataTypes.INT) {
+int size = pageSize * DataTypes.INT.getSizeInBytes();
+for (int i = 0; i < size; i += DataTypes.INT.getSizeInBytes()) {
+ vector.putFloat(rowId++,
+ (ByteUtil.toIntLittleEndian(pageData, pageIndex + i) /
floatFactor));
+}
+pageInde
[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI
Karan980 commented on pull request #3876: URL: https://github.com/apache/carbondata/pull/3876#issuecomment-673388953 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI
CarbonDataQA1 commented on pull request #3876: URL: https://github.com/apache/carbondata/pull/3876#issuecomment-673383030 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1969/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3876: TestingCI
CarbonDataQA1 commented on pull request #3876: URL: https://github.com/apache/carbondata/pull/3876#issuecomment-673382375 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3708/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3778: [WIP][CARBONDATA-3916] Support array complex type with SI
QiangCai commented on pull request #3778: URL: https://github.com/apache/carbondata/pull/3778#issuecomment-673324648 please rebase This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3874: [CARBONDATA-3931]Fix Secondary index with index column as DateType giving wrong results
asfgit closed pull request #3874: URL: https://github.com/apache/carbondata/pull/3874 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3874: [CARBONDATA-3931]Fix Secondary index with index column as DateType giving wrong results
QiangCai commented on pull request #3874: URL: https://github.com/apache/carbondata/pull/3874#issuecomment-673319957 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3877: [CARBONDATA-3889] Cleanup duplicated code in carbondata-spark module
asfgit closed pull request #3877: URL: https://github.com/apache/carbondata/pull/3877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] kevinjmh commented on pull request #3877: [CARBONDATA-3889] Cleanup duplicated code in carbondata-spark module
kevinjmh commented on pull request #3877: URL: https://github.com/apache/carbondata/pull/3877#issuecomment-673311567 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3877: [CARBONDATA-3889] Cleanup duplicated code in carbondata-spark module
CarbonDataQA1 commented on pull request #3877: URL: https://github.com/apache/carbondata/pull/3877#issuecomment-673310278 Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/1968/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] Karan980 commented on pull request #3876: TestingCI
Karan980 commented on pull request #3876: URL: https://github.com/apache/carbondata/pull/3876#issuecomment-673309930 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3877: [CARBONDATA-3889] Cleanup duplicated code in carbondata-spark module
CarbonDataQA1 commented on pull request #3877: URL: https://github.com/apache/carbondata/pull/3877#issuecomment-673303190 Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3707/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
