[GitHub] carbondata pull request #2730: [WIP] Performance improvements for Fileformat...
Github user ravipesala closed the pull request at: https://github.com/apache/carbondata/pull/2730 ---
[GitHub] carbondata pull request #2730: [WIP] Performance improvements for Fileformat...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2730#discussion_r224746003
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/directread/ColumnarVectorWrapperDirectWithDeleteDelta.java
---
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.result.vector.impl.directread;
+
+import java.math.BigDecimal;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+
+class ColumnarVectorWrapperDirectWithDeleteDelta extends
AbstractCarbonColumnarVector {
+
+ private BitSet deletedRows;
+
+ private BitSet nullBits;
+
+ private int counter;
+
+ private CarbonColumnVector columnVector;
--- End diff --
`columnVector` can be made protected and moved to abstract class
---
[GitHub] carbondata pull request #2730: [WIP] Performance improvements for Fileformat...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2730#discussion_r224738600
--- Diff:
core/src/main/java/org/apache/carbondata/core/scan/result/vector/impl/directread/ColumnarVectorWrapperDirectWithDeleteDeltaAndInvertedIndex.java
---
@@ -0,0 +1,228 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.scan.result.vector.impl.directread;
+
+import java.math.BigDecimal;
+import java.util.BitSet;
+
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.datatype.DecimalType;
+import org.apache.carbondata.core.scan.result.vector.CarbonColumnVector;
+import
org.apache.carbondata.core.scan.result.vector.impl.CarbonColumnVectorImpl;
+
+class ColumnarVectorWrapperDirectWithDeleteDeltaAndInvertedIndex
+extends AbstractCarbonColumnarVector implements ConvertableVector {
--- End diff --
AbstractCarbonColumnarVector is already implementing ConvertableVector
...so it can be removed from here
---
[GitHub] carbondata pull request #2730: [WIP] Performance improvements for Fileformat...
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2730#discussion_r224776123
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/datatype/DecimalConverterFactory.java
---
@@ -95,6 +100,51 @@ public int getSizeInBytes() {
return BigDecimal.valueOf((Long) valueToBeConverted, scale);
}
+@Override public void fillVector(Object valuesToBeConverted, int size,
ColumnVectorInfo info,
+BitSet nullBitset) {
+ // TODO we need to find way to directly set to vector with out
conversion. This way is very
+ // inefficient.
+ CarbonColumnVector vector = info.vector;
+ int precision = info.measure.getMeasure().getPrecision();
+ if (valuesToBeConverted instanceof byte[]) {
+byte[] data = (byte[]) valuesToBeConverted;
+for (int i = 0; i < size; i++) {
+ if (nullBitset.get(i)) {
+vector.putNull(i);
+ } else {
+vector.putDecimal(i, BigDecimal.valueOf(data[i], scale),
precision);
+ }
+}
+ } else if (valuesToBeConverted instanceof short[]) {
+short[] data = (short[]) valuesToBeConverted;
+for (int i = 0; i < size; i++) {
+ if (nullBitset.get(i)) {
+vector.putNull(i);
+ } else {
+vector.putDecimal(i, BigDecimal.valueOf(data[i], scale),
precision);
+ }
+}
+ } else if (valuesToBeConverted instanceof int[]) {
+int[] data = (int[]) valuesToBeConverted;
+for (int i = 0; i < size; i++) {
+ if (nullBitset.get(i)) {
+vector.putNull(i);
+ } else {
+vector.putDecimal(i, BigDecimal.valueOf(data[i], scale),
precision);
+ }
+}
+ } else if (valuesToBeConverted instanceof long[]) {
+long[] data = (long[]) valuesToBeConverted;
+for (int i = 0; i < size; i++) {
+ if (nullBitset.get(i)) {
+vector.putNull(i);
+ } else {
+vector.putDecimal(i, BigDecimal.valueOf(data[i], scale),
precision);
+ }
+}
--- End diff --
Alter scenario where precision and scale can be changed is not handled
here...please check for that case and handle...you can take reference from the
`DecimalMeasureVectorFiller`
---
[GitHub] carbondata pull request #2730: [WIP] Performance improvements for Fileformat...
GitHub user ravipesala opened a pull request: https://github.com/apache/carbondata/pull/2730 [WIP] Performance improvements for Fileformat and Presto Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/incubator-carbondata perf-poc-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2730.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2730 commit 3a98aa6cf0bdee71f078c9cac6676177bdbee6d4 Author: ravipesala Date: 2018-09-04T11:07:17Z Supported full scan queries commit 33a9105eaac6134b0aad1022b4d0d82c02230d4b Author: ravipesala Date: 2018-09-06T15:59:37Z Supported filter queries commit 786c754903e99e69fe4905195052d84a24527a01 Author: ravipesala Date: 2018-09-08T12:47:59Z Added new vector wrapper commit 633ff438cb784ef3354d6d2bfcff5fbe27c02281 Author: ravipesala Date: 2018-09-14T10:54:27Z After Rebase commit dce0f297337dec2fed914d53386bc386ec9cd862 Author: ravipesala Date: 2018-09-17T11:41:41Z Refactored ---
