svn commit: r39878 - /dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/
Author: haibow Date: Tue Jun 2 06:49:10 2020 New Revision: 39878 Log: Update apache-pinot-incubating-0.4.0-rc2 Added: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/ dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz (with props) dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.asc dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.sha512 dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz (with props) dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz.asc dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz.sha512 Added: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz == Binary file - no diff available. Propchange: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.asc == --- dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.asc (added) +++ dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.asc Tue Jun 2 06:49:10 2020 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEyH5xT96Wd9MlHHp5bMFppvwZxHAFAl7V8TIACgkQbMFppvwZ +xHCCOxAAgcS8toKtgiR1dxrMoVB4DYt2BF0jPsONoxRb2yiotpncfJPw9V++85f2 +eMkFE/4crqXhnvvjnhZpbOqA4/mds0fwwTMz2F/nLhavB70NZqNbx5yYo4eqXJj4 +NEn/go744cDzq9gxT/0OIKCffEVYJqyQag/Ww0yb8KRkZO1wth15k2p/xuum4Qet +4ON5lfa6FaIy/pGrdhuT353GnNgMjCwbQh7By82Ib+hyhMcoYvXSi/AnpmoeWc1n +6SC4561NwJlN6YmPVoJkXUaGaaPriXgXdHpy2llCYULt4Q/HE/e7cgV1Hy/BTydG +GeKWdg8pzqkcK0h/JFJdM49RbneW8wJbQOLTl/XcgMugqkUCm3to5YR0cncVNpCH +DHDLUSzkKtdhqNCLeG+0tCBJ5nKY9kz0dlYW5yGWXiRoYyTeMyNb/KPcTzL3a2fv +r7AiQ7/mnB0x5zYlc4Dmed2+CF+HYEVivvyzOKDS+N8fhbjIUcidsMa3m5ac9gi8 +12uR8WyCd5n3wrwQl3a76KSkO9Q+VISbPZZ3ovN6ymOOuVrK03lWzcSwWA8g0XNl +AJJMQb4R1QTucU2ApwXglfSUD9r5y9unUU+XyGgfRNxrhOc6klnWhyWhMYo09Vdq +GJgKLgHpUiMI65PHSpYVF3AORE68ma0YKx/NZZep41Gr3mn76ns= +=fa1X +-END PGP SIGNATURE- Added: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.sha512 == --- dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.sha512 (added) +++ dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-bin.tar.gz.sha512 Tue Jun 2 06:49:10 2020 @@ -0,0 +1 @@ +155494c13501bad1adb626cf204079d024e72b1ce700ab55c8de2e7f4a7166b77f3715b9af9ff2e077681ca11e4cc87987433cd56572226688999dc9e413 \ No newline at end of file Added: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz == Binary file - no diff available. Propchange: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz.asc == --- dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz.asc (added) +++ dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/apache-pinot-incubating-0.4.0-src.tar.gz.asc Tue Jun 2 06:49:10 2020 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEyH5xT96Wd9MlHHp5bMFppvwZxHAFAl7V8TEACgkQbMFppvwZ +xHACDBAAjCsrfhSmaw1fKZ0ot5SE1FHVP00A6Sa2y5csVW7XpAklaefFvzR8LGLZ ++OpMXH7/BJn/pf054UnHFCntWQBPyZcZfNyVSxxT6h39F1JDE8Rbe4y7VAsWwn8a +YaCBbW/U9gp69OTRHyKHerTbDe89KczZD02xsgxiW6N4lxvyGEcE9DBtzJwhuc35 +pp8z3s2wYFv45j4myuyU7KE3NORB73dqsqkV0Y7nccilLHGR96GNEFeBaGRnUMip +T2IfXge+e6QSfKhii/wYKOs2+dOPcdflV9rMiLqsL3BH6ajAPGnyc2+poAa1eX0e +wf+75yMg8nfenyFnaE1iQo5cnbd2354fowEMprJj5M44YcMAq2U6xPbbCBWqGaqO +rotrIRAy/K6yxovJQarGtgyqpp2pTXP0k4MlcwdzhZfEmp8pZrOpwCCaGQ0/fn/U +Qcr//wTsqA576Ha1eE3YoLZN+7MDLH9H6EpOpz0vMJrsB3XQbz0UfTcCnD5eyJ98 +qr9If90rJXbkNdttr1ODbz6TqHnwjAM8w0N1Y8WPzhyOn5/+U8Y3MHmskMEvRw2w +tKan6DmogpJHz4NnklDzcV+mti1W8IBs+/kyUItopjf++xmhMDcqCZPZWTs6/omQ +ond7Vh1tjfG6Lzdj7lA5NE+8WdDWC7pICiwz2SUdgF/UBmStaLI= +=uGOJ +-END PGP SIGNATURE- Added: dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc2/a
[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5444: Enhance and simplify the filtering
chenboat commented on a change in pull request #5444:
URL: https://github.com/apache/incubator-pinot/pull/5444#discussion_r433647278
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/AndDocIdIterator.java
##
@@ -18,112 +18,52 @@
*/
package org.apache.pinot.core.operator.dociditerators;
-import java.util.Arrays;
import org.apache.pinot.core.common.BlockDocIdIterator;
import org.apache.pinot.core.common.Constants;
-// TODO: Optimize this
public final class AndDocIdIterator implements BlockDocIdIterator {
Review comment:
1.javadoc?
2. comments about thread safety. is it safe/advisable to use this iterator
in more than one thread?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5444: Enhance and simplify the filtering
chenboat commented on a change in pull request #5444:
URL: https://github.com/apache/incubator-pinot/pull/5444#discussion_r433642778
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/ArrayBasedDocIdIterator.java
##
@@ -36,37 +35,15 @@ public ArrayBasedDocIdIterator(int[] docIds, int
searchableLength) {
@Override
public int next() {
-if (_currentDocId == Constants.EOF) {
- return Constants.EOF;
-}
-if (++_currentIndex == _searchableLength) {
- _currentDocId = Constants.EOF;
+if (_nextIndex < _searchableLength) {
+ return _docIds[_nextIndex++];
Review comment:
what happens if _nextIndex >= docIds.length? should we have a array
length check?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] chenboat commented on a change in pull request #5444: Enhance and simplify the filtering
chenboat commented on a change in pull request #5444:
URL: https://github.com/apache/incubator-pinot/pull/5444#discussion_r433641032
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/AndDocIdIterator.java
##
@@ -18,112 +18,52 @@
*/
package org.apache.pinot.core.operator.dociditerators;
-import java.util.Arrays;
import org.apache.pinot.core.common.BlockDocIdIterator;
import org.apache.pinot.core.common.Constants;
-// TODO: Optimize this
public final class AndDocIdIterator implements BlockDocIdIterator {
- public final BlockDocIdIterator[] docIdIterators;
- public ScanBasedDocIdIterator[] scanBasedDocIdIterators;
- public final int[] docIdPointers;
- public boolean reachedEnd = false;
- public int currentDocId = -1;
- int currentMax = -1;
- private boolean hasScanBasedIterators;
+ public final BlockDocIdIterator[] _docIdIterators;
- public AndDocIdIterator(BlockDocIdIterator[] blockDocIdIterators) {
-int numIndexBasedIterators = 0;
-int numScanBasedIterators = 0;
-for (int i = 0; i < blockDocIdIterators.length; i++) {
- if (blockDocIdIterators[i] instanceof IndexBasedDocIdIterator) {
-numIndexBasedIterators = numIndexBasedIterators + 1;
- } else if (blockDocIdIterators[i] instanceof ScanBasedDocIdIterator) {
-numScanBasedIterators = numScanBasedIterators + 1;
- }
-}
-// if we have at least one index based then do intersection based on index
based only, and then
-// check if matching docs apply on scan based iterator
-if (numIndexBasedIterators > 0 && numScanBasedIterators > 0) {
- hasScanBasedIterators = true;
- int nonScanIteratorsSize = blockDocIdIterators.length -
numScanBasedIterators;
- this.docIdIterators = new BlockDocIdIterator[nonScanIteratorsSize];
- this.scanBasedDocIdIterators = new
ScanBasedDocIdIterator[numScanBasedIterators];
- int nonScanBasedIndex = 0;
- int scanBasedIndex = 0;
- for (int i = 0; i < blockDocIdIterators.length; i++) {
-if (blockDocIdIterators[i] instanceof ScanBasedDocIdIterator) {
- this.scanBasedDocIdIterators[scanBasedIndex++] =
(ScanBasedDocIdIterator) blockDocIdIterators[i];
-} else {
- this.docIdIterators[nonScanBasedIndex++] = blockDocIdIterators[i];
-}
- }
-} else {
- hasScanBasedIterators = false;
- this.docIdIterators = blockDocIdIterators;
-}
-this.docIdPointers = new int[docIdIterators.length];
-Arrays.fill(docIdPointers, -1);
- }
+ private int _nextDocId = 0;
- @Override
- public int advance(int targetDocId) {
-if (currentDocId == Constants.EOF) {
- return currentDocId;
-}
-if (currentDocId >= targetDocId) {
- return currentDocId;
-}
-// next() method will always increment currentMax by 1.
-currentMax = targetDocId - 1;
-return next();
+ public AndDocIdIterator(BlockDocIdIterator[] docIdIterators) {
+_docIdIterators = docIdIterators;
}
@Override
public int next() {
-if (currentDocId == Constants.EOF) {
- return currentDocId;
-}
-currentMax = currentMax + 1;
-// always increment the pointer to current max, when this is called first
time, every one will
-// be set to start of posting list.
-for (int i = 0; i < docIdIterators.length; i++) {
- docIdPointers[i] = docIdIterators[i].advance(currentMax);
- if (docIdPointers[i] == Constants.EOF) {
-reachedEnd = true;
-currentMax = Constants.EOF;
-break;
- }
- if (docIdPointers[i] > currentMax) {
-currentMax = docIdPointers[i];
-if (i > 0) {
- // we need to advance all pointer since we found a new max
- i = -1;
-}
+int maxDocId = _nextDocId;
Review comment:
i think a better name could be candidateDocId because the while loop()
is searching for the next doc Id exists in all iterators. maxDocId to many ppl
mean the maximum doc id in the iterator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] chenboat commented on pull request #5478: Add a new table config field for peer segment download.
chenboat commented on pull request #5478:
URL: https://github.com/apache/incubator-pinot/pull/5478#issuecomment-637276829
> Please add validation code ("Must be http or https") in
TableConfigUtils.validate() thanks
> Also, add more documentation on the new config variable (at least a
pointer to the design doc).
> Please label the PR marking it for release notes.
> Make sure you add documentation on the config in gitbook (after you merge
this PR)
Done except that the last point about documentation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5440: Add GenericTransformFunction wrapper for simple ScalarFunctions
Jackie-Jiang commented on a change in pull request #5440:
URL: https://github.com/apache/incubator-pinot/pull/5440#discussion_r433616803
##
File path: pinot-common/pom.xml
##
@@ -33,6 +33,7 @@
https://pinot.apache.org/
${basedir}/..
+0.9.11
Review comment:
What I mean is moving the dependency and version into
`dependencyManagement` in the root pom file. You may refer to other
dependencies in pinot-common
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5440: Add GenericTransformFunction wrapper for simple ScalarFunctions
Jackie-Jiang commented on a change in pull request #5440:
URL: https://github.com/apache/incubator-pinot/pull/5440#discussion_r433616232
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java
##
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.FunctionInfo;
+import org.apache.pinot.common.function.FunctionInvoker;
+import org.apache.pinot.core.common.DataSource;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class ScalarTransformFunctionWrapper extends BaseTransformFunction {
+
+ FunctionInvoker _functionInvoker;
+ String _name;
+ Object[] _args;
+ List _nonLiteralArgIndices;
+ List _nonLiteralArgType;
+ List _nonLiteralTransformFunction;
+ TransformResultMetadata _transformResultMetadata;
+ String[] _stringResult;
+ int[] _integerResult;
+ float[] _floatResult;
+ double[] _doubleResult;
+ long[] _longResult;
+
+ public ScalarTransformFunctionWrapper() {
+_nonLiteralArgIndices = new ArrayList<>();
+_nonLiteralArgType = new ArrayList<>();
+_nonLiteralTransformFunction = new ArrayList<>();
+ }
+
+ @Override
+ public String getName() {
+return _name;
+ }
+
+ public void setFunction(String functionName, FunctionInfo info)
+ throws Exception {
+_name = functionName;
+_functionInvoker = new FunctionInvoker(info);
+ }
+
+ @Override
+ public void init(List arguments, Map
dataSourceMap) {
+Preconditions.checkArgument(arguments.size() ==
_functionInvoker.getParameterTypes().length,
+"The number of arguments are not same for scalar function and
transform function: %s", getName());
+
+_args = new Object[arguments.size()];
+for (int i = 0; i < arguments.size(); i++) {
+ TransformFunction function = arguments.get(i);
+ if (function instanceof LiteralTransformFunction) {
+String literal = ((LiteralTransformFunction) function).getLiteral();
+Class paramType = _functionInvoker.getParameterTypes()[i];
+switch (paramType.getTypeName()) {
+ case "java.lang.Integer":
+_args[i] = Integer.parseInt(literal);
+break;
+ case "java.lang.String":
+_args[i] = literal;
+break;
+ case "java.lang.Double":
+_args[i] = Double.valueOf(literal);
+break;
+ case "java.lang.Float":
+_args[i] = Float.valueOf(literal);
+break;
+ case "java.lang.Long":
+_args[i] = Long.valueOf(literal);
+break;
+ default:
+throw new RuntimeException(
+"Unsupported data type " + paramType.getTypeName() + "for
transform function " + getName());
+}
+ } else {
+_nonLiteralArgIndices.add(i);
+_nonLiteralTransformFunction.add(function);
+Class paramType = _functionInvoker.getParameterTypes()[i];
+
+switch (paramType.getTypeName()) {
+ case "java.lang.Integer":
+_nonLiteralArgType.add(FieldSpec.DataType.INT);
+break;
+ case "java.lang.String":
+_nonLiteralArgType.add(FieldSpec.DataType.STRING);
+break;
+ case "java.lang.Double":
+_nonLiteralArgType.add(FieldSpec.DataType.DOUBLE);
+break;
+ case "java.lang.Float":
+_nonLiteralArgType.add(FieldSpec.DataType.FLOAT);
+break;
+ case "java.lang.Long":
+_nonLiteralArgType.add(FieldSpec.DataType.LONG);
+break;
+ default:
+throw new RuntimeException(
+"Unsupported data type " + paramTy
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5440: Add GenericTransformFunction wrapper for simple ScalarFunctions
Jackie-Jiang commented on a change in pull request #5440:
URL: https://github.com/apache/incubator-pinot/pull/5440#discussion_r433615835
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TransformFunctionFactory.java
##
@@ -112,13 +115,24 @@ public static TransformFunction
get(TransformExpressionTree expression, Map transformFunctionClass =
TRANSFORM_FUNCTION_MAP.get(functionName);
+Class transformFunctionClass;
+FunctionInfo functionInfo = null;
+if (FunctionRegistry.containsFunctionByName(functionName)) {
+ transformFunctionClass = ScalarTransformFunctionWrapper.class;
+ functionInfo = FunctionRegistry.getFunctionByName(functionName);
+} else {
+ transformFunctionClass = TRANSFORM_FUNCTION_MAP.get(functionName);
+}
+
if (transformFunctionClass == null) {
throw new BadQueryRequestException("Unsupported transform function:
" + functionName);
}
try {
transformFunction = transformFunctionClass.newInstance();
-} catch (InstantiationException | IllegalAccessException e) {
+ if (functionInfo != null) {
+((ScalarTransformFunctionWrapper)
transformFunction).setFunction(functionName, functionInfo);
+ }
+} catch (Exception e) {
Review comment:
Never mind, I thought this try-catch is around line 128
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch release-0.4.0-rc-2 updated: [maven-release-plugin] prepare for next development iteration
This is an automated email from the ASF dual-hosted git repository. haibow pushed a commit to branch release-0.4.0-rc-2 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/release-0.4.0-rc-2 by this push: new 28efe1d [maven-release-plugin] prepare for next development iteration 28efe1d is described below commit 28efe1d4aa7c052ab1e5c4de956371a8dc4fdecc Author: Haibo Wang AuthorDate: Mon Jun 1 21:38:19 2020 -0700 [maven-release-plugin] prepare for next development iteration --- pinot-broker/pom.xml | 2 +- pinot-clients/pinot-java-client/pom.xml | 2 +- pinot-clients/pom.xml | 2 +- pinot-common/pom.xml | 2 +- pinot-controller/pom.xml | 2 +- pinot-core/pom.xml| 2 +- pinot-distribution/pom.xml| 2 +- pinot-integration-tests/pom.xml | 2 +- pinot-minion/pom.xml | 2 +- pinot-perf/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-common/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pom.xml| 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-spark/pom.xml | 2 +- .../pinot-batch-ingestion/pinot-batch-ingestion-standalone/pom.xml| 2 +- pinot-plugins/pinot-batch-ingestion/pom.xml | 2 +- .../pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml | 2 +- .../v0_deprecated/pinot-ingestion-common/pom.xml | 2 +- pinot-plugins/pinot-batch-ingestion/v0_deprecated/pinot-spark/pom.xml | 2 +- pinot-plugins/pinot-batch-ingestion/v0_deprecated/pom.xml | 2 +- pinot-plugins/pinot-file-system/pinot-adls/pom.xml| 2 +- pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 2 +- pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml| 2 +- pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 2 +- pinot-plugins/pinot-file-system/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-avro-base/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-avro/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-confluent-avro/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-csv/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-json/pom.xml | 2 +- pinot-plugins/pinot-input-format/pinot-orc/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-parquet/pom.xml| 2 +- pinot-plugins/pinot-input-format/pinot-thrift/pom.xml | 2 +- pinot-plugins/pinot-input-format/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pinot-kafka-base/pom.xml | 2 +- pinot-plugins/pinot-stream-ingestion/pom.xml | 2 +- pinot-plugins/pom.xml | 2 +- pinot-server/pom.xml | 2 +- pinot-spi/pom.xml | 2 +- pinot-tools/pom.xml | 2 +- pom.xml | 4 ++-- 42 files changed, 43 insertions(+), 43 deletions(-) diff --git a/pinot-broker/pom.xml b/pinot-broker/pom.xml index 2fd885d..a884630 100644 --- a/pinot-broker/pom.xml +++ b/pinot-broker/pom.xml @@ -24,7 +24,7 @@ pinot org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT .. pinot-broker diff --git a/pinot-clients/pinot-java-client/pom.xml b/pinot-clients/pinot-java-client/pom.xml index 615c5e9..60177d7 100644 --- a/pinot-clients/pinot-java-client/pom.xml +++ b/pinot-clients/pinot-java-client/pom.xml @@ -24,7 +24,7 @@ pinot-clients org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT .. pinot-java-client diff --git a/pinot-clients/pom.xml b/pinot-clients/pom.xml index e05e7ec..902f54f 100644 --- a/pinot-clients/pom.xml +++ b/pinot-clients/pom.xml @@ -24,7 +24,7 @@ pinot org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT .. pinot-clients diff --git a/pinot-common/pom.xml b/pinot-common/pom.xml index db2d28d..7f2a7fc 100644 --- a/pinot-common/pom.xml +++ b/pinot-common/pom.xml @@ -24,7 +24,7 @@ pinot org.apache.pinot -0.4.0 +0.5.0-SNAPSHOT ..
[incubator-pinot] annotated tag release-0.4.0-rc2 updated (8355d2e -> 88accfc)
This is an automated email from the ASF dual-hosted git repository. haibow pushed a change to annotated tag release-0.4.0-rc2 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag release-0.4.0-rc2 was modified! *** from 8355d2e (commit) to 88accfc (tag) tagging 8355d2e0e489a8d127f2e32793671fba505628a8 (commit) by Haibo Wang on Mon Jun 1 21:38:14 2020 -0700 - Log - [maven-release-plugin] copy for tag release-0.4.0-rc2 --- No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch release-0.4.0-rc-2 updated: [maven-release-plugin] prepare release release-0.4.0-rc2
This is an automated email from the ASF dual-hosted git repository.
haibow pushed a commit to branch release-0.4.0-rc-2
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/release-0.4.0-rc-2 by this
push:
new 8355d2e [maven-release-plugin] prepare release release-0.4.0-rc2
8355d2e is described below
commit 8355d2e0e489a8d127f2e32793671fba505628a8
Author: Haibo Wang
AuthorDate: Mon Jun 1 21:38:00 2020 -0700
[maven-release-plugin] prepare release release-0.4.0-rc2
---
pinot-broker/pom.xml | 5 ++---
pinot-clients/pinot-java-client/pom.xml | 5 ++---
pinot-clients/pom.xml | 6 ++
pinot-common/pom.xml | 5 ++---
pinot-controller/pom.xml | 5 ++---
pinot-core/pom.xml| 5 ++---
pinot-distribution/pom.xml| 7 +++
pinot-integration-tests/pom.xml | 5 ++---
pinot-minion/pom.xml | 5 ++---
pinot-perf/pom.xml| 5 ++---
.../pinot-batch-ingestion/pinot-batch-ingestion-common/pom.xml| 6 ++
.../pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pom.xml| 6 ++
.../pinot-batch-ingestion/pinot-batch-ingestion-spark/pom.xml | 6 ++
.../pinot-batch-ingestion-standalone/pom.xml | 6 ++
pinot-plugins/pinot-batch-ingestion/pom.xml | 6 ++
.../pinot-batch-ingestion/v0_deprecated/pinot-hadoop/pom.xml | 7 +++
.../v0_deprecated/pinot-ingestion-common/pom.xml | 6 ++
.../pinot-batch-ingestion/v0_deprecated/pinot-spark/pom.xml | 7 +++
pinot-plugins/pinot-batch-ingestion/v0_deprecated/pom.xml | 6 ++
pinot-plugins/pinot-file-system/pinot-adls/pom.xml| 5 ++---
pinot-plugins/pinot-file-system/pinot-gcs/pom.xml | 6 ++
pinot-plugins/pinot-file-system/pinot-hdfs/pom.xml| 5 ++---
pinot-plugins/pinot-file-system/pinot-s3/pom.xml | 6 ++
pinot-plugins/pinot-file-system/pom.xml | 6 ++
pinot-plugins/pinot-input-format/pinot-avro-base/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-avro/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-confluent-avro/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-csv/pom.xml| 5 ++---
pinot-plugins/pinot-input-format/pinot-json/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pinot-orc/pom.xml| 6 ++
pinot-plugins/pinot-input-format/pinot-parquet/pom.xml| 5 ++---
pinot-plugins/pinot-input-format/pinot-thrift/pom.xml | 5 ++---
pinot-plugins/pinot-input-format/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pinot-kafka-0.9/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pinot-kafka-2.0/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pinot-kafka-base/pom.xml | 6 ++
pinot-plugins/pinot-stream-ingestion/pom.xml | 6 ++
pinot-plugins/pom.xml | 8 +++-
pinot-server/pom.xml | 5 ++---
pinot-spi/pom.xml | 5 ++---
pinot-tools/pom.xml | 5 ++---
pom.xml | 7 +++
42 files changed, 89 insertions(+), 149 deletions(-)
diff --git a/pinot-broker/pom.xml b/pinot-broker/pom.xml
index d9e48e1..2fd885d 100644
--- a/pinot-broker/pom.xml
+++ b/pinot-broker/pom.xml
@@ -19,13 +19,12 @@
under the License.
-->
-http://www.w3.org/2001/XMLSchema-instance";
xmlns="http://maven.apache.org/POM/4.0.0";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
4.0.0
pinot
org.apache.pinot
-${revision}${sha1}
+0.4.0
..
pinot-broker
diff --git a/pinot-clients/pinot-java-client/pom.xml
b/pinot-clients/pinot-java-client/pom.xml
index 6a98e3d..615c5e9 100644
--- a/pinot-clients/pinot-java-client/pom.xml
+++ b/pinot-clients/pinot-java-client/pom.xml
@@ -19,13 +19,12 @@
under the License.
-->
-http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
ht
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5483: Add ServerQuery to replace BrokerRequest in server execution engine
Jackie-Jiang commented on a change in pull request #5483:
URL: https://github.com/apache/incubator-pinot/pull/5483#discussion_r433615031
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/request/v2/ServerQuery.java
##
@@ -0,0 +1,181 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.v2;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.BrokerRequest;
+
+
+public class ServerQuery {
Review comment:
This object can have some Server specific optimizations or pre-computed
result, which might not apply to Broker. For example, we don't need to compile
the filter for broker. We can add a separate BrokerQuery which only contain the
information that Broker needs.
Expression, Function are common and shared for both Broker and Server.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 02/02: Initialize _aggregateMetric in constructor (#5472)
This is an automated email from the ASF dual-hosted git repository.
haibow pushed a commit to branch release-0.4.0-rc-2
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 9e0bb880366fc3e895c097b37e0accbeb28ab586
Author: Sidd
AuthorDate: Sun May 31 15:35:46 2020 -0700
Initialize _aggregateMetric in constructor (#5472)
Co-authored-by: Siddharth Teotia
---
.../apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
index 0e24664..b3a2fed 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
@@ -171,6 +171,7 @@ public class MutableSegmentImpl implements MutableSegment {
_partitionFunction = config.getPartitionFunction();
_partitionId = config.getPartitionId();
_nullHandlingEnabled = config.isNullHandlingEnabled();
+_aggregateMetrics = config.aggregateMetrics();
Collection allFieldSpecs = _schema.getAllFieldSpecs();
List physicalFieldSpecs = new ArrayList<>(allFieldSpecs.size());
@@ -344,7 +345,8 @@ public class MutableSegmentImpl implements MutableSegment {
// if the column is part of noDictionary set from table config
if (fieldSpec instanceof DimensionFieldSpec && _aggregateMetrics &&
(dataType == FieldSpec.DataType.STRING ||
dataType == FieldSpec.DataType.BYTES)) {
-_logger.info("Not creating dictionary in consuming segment for column
{} of type {}", column, dataType.toString());
+_logger.info("Aggregate metrics is enabled. Will create dictionary in
consuming segment for column {} of type {}",
+column, dataType.toString());
return false;
}
// So don't create dictionary if the column is member of noDictionary,
is single-value
@@ -857,7 +859,6 @@ public class MutableSegmentImpl implements MutableSegment {
*/
private IdMap
enableMetricsAggregationIfPossible(RealtimeSegmentConfig config,
Set noDictionaryColumns) {
-_aggregateMetrics = config.aggregateMetrics();
if (!_aggregateMetrics) {
_logger.info("Metrics aggregation is disabled.");
return null;
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch release-0.4.0-rc-2 created (now 9e0bb88)
This is an automated email from the ASF dual-hosted git repository. haibow pushed a change to branch release-0.4.0-rc-2 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 9e0bb88 Initialize _aggregateMetric in constructor (#5472) This branch includes the following new commits: new fe3b98e Two changes: (#5470) new 9e0bb88 Initialize _aggregateMetric in constructor (#5472) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/02: Two changes: (#5470)
This is an automated email from the ASF dual-hosted git repository.
haibow pushed a commit to branch release-0.4.0-rc-2
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit fe3b98ea8d2283d4b67b0bdb8622cbf4490f2cc1
Author: Sidd
AuthorDate: Sat May 30 19:21:23 2020 -0700
Two changes: (#5470)
(1) PR https://github.com/apache/incubator-pinot/pull/5256
added support for deriving num docs per chunk for var byte
raw index create from column length. This was specifically
done as part of supporting text blobs. For use cases that
don't want this feature and are high QPS, see a negative
impact since size of chunk increases (earlier value
of numDocsPerChunk was hardcoded to 1000) and based on the
access pattern we might end up uncompressing a bigger chunk to get values
for a set of docIds. We have made this change configurable.
So the default behaviour is same as old (1000 docs per chunk)
(2) PR https://github.com/apache/incubator-pinot/pull/4791
added support for noDict for STRING/BYTES in consuming segments.
There is a particular impact of this change on the use cases
that have set noDict on their STRING dimension columns for other performance
reasons and also want metricsAggregation. These use cases don't get to
aggregateMetrics because the new implementation was able to honor their
table config setting of noDict on STRING/BYTES. Without metrics aggregation,
memory pressure increases. So to continue aggregating metrics for such
cases,
we will create dictionary even if the column is part of noDictionary set
from table config.
Co-authored-by: Siddharth Teotia
---
.../generator/SegmentGeneratorConfig.java | 15 +++
.../indexsegment/mutable/MutableSegmentImpl.java | 29 +++---
.../creator/impl/SegmentColumnarIndexCreator.java | 22 ++--
.../fwd/SingleValueVarByteRawIndexCreator.java | 10 +++-
.../defaultcolumn/BaseDefaultColumnHandler.java| 3 ++-
.../apache/pinot/spi/config/table/FieldConfig.java | 1 +
6 files changed, 72 insertions(+), 8 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
index 59531fe..9af9c16 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/generator/SegmentGeneratorConfig.java
@@ -102,6 +102,9 @@ public class SegmentGeneratorConfig {
private boolean _skipTimeValueCheck = false;
private boolean _nullHandlingEnabled = false;
+ // constructed from FieldConfig
+ private Map> _columnProperties = new HashMap<>();
+
@Deprecated
public SegmentGeneratorConfig() {
}
@@ -174,12 +177,24 @@ public class SegmentGeneratorConfig {
_invertedIndexCreationColumns =
indexingConfig.getInvertedIndexColumns();
}
+ List fieldConfigList = tableConfig.getFieldConfigList();
+ if (fieldConfigList != null) {
+for (FieldConfig fieldConfig : fieldConfigList) {
+ _columnProperties.put(fieldConfig.getName(),
fieldConfig.getProperties());
+}
+ }
+
extractTextIndexColumnsFromTableConfig(tableConfig);
_nullHandlingEnabled = indexingConfig.isNullHandlingEnabled();
}
}
+ @Nonnull
+ public Map> getColumnProperties() {
+return _columnProperties;
+ }
+
/**
* Set time column details using the given time column
*/
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
index 07e5ec9..0e24664 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/indexsegment/mutable/MutableSegmentImpl.java
@@ -219,7 +219,7 @@ public class MutableSegmentImpl implements MutableSegment {
FieldSpec.DataType dataType = fieldSpec.getDataType();
boolean isFixedWidthColumn = dataType.isFixedWidth();
int forwardIndexColumnSize = -1;
- if (isNoDictionaryColumn(noDictionaryColumns, invertedIndexColumns,
textIndexColumns, fieldSpec, column)) {
+ if (isNoDictionaryColumn(noDictionaryColumns, invertedIndexColumns,
fieldSpec, column)) {
// no dictionary
// each forward index entry will be equal to size of data for that row
// For INT, LONG, FLOAT, DOUBLE it is equal to the number of fixed
bytes used to store the value,
@@ -329,9 +329,30 @@ public class MutableSegmentImpl implements MutableSegment {
* @return true if column is no-dictionary, false if dictionary encoded
*/
private boolean isNoDictionaryColumn(Set noDictionaryColumns,
Set in
[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #5483: Add ServerQuery to replace BrokerRequest in server execution engine
kishoreg commented on a change in pull request #5483:
URL: https://github.com/apache/incubator-pinot/pull/5483#discussion_r433593063
##
File path:
pinot-common/src/main/java/org/apache/pinot/common/request/v2/ServerQuery.java
##
@@ -0,0 +1,181 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.request.v2;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import org.apache.pinot.common.request.BrokerRequest;
+
+
+public class ServerQuery {
Review comment:
Why call this ServerQuery?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar opened a new pull request #5484: Extract all fields if fieldsToRead is null/empty
npawar opened a new pull request #5484: URL: https://github.com/apache/incubator-pinot/pull/5484 ## Description In the RecordExtractors, if `fieldsToRead` is null/empty, extract all fields from the record. The fieldsToRead can become as the "includeList". In the future we can introduce an "excludeList". This will all be useful once we start doing advanced flattening (e.g. flatten everything, include x,y exclude p,q,r) Additionally, removing some sample groovy schemas, as we stopped using them when the refactoring to exclude Schema from RecordReaders/Decoders was done. ## Upgrade Notes NA ## Documentation NA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5483: Add ServerQuery to replace BrokerRequest in server execution engine
Jackie-Jiang opened a new pull request #5483: URL: https://github.com/apache/incubator-pinot/pull/5483 Motivation: For historic reason, BrokerRequest does not support expressions natively. In order to support expressions (transform), the current solution is to store the expressions as String and compile them when constructing the operators. The problem with this approach is that the same compilation is performed multiple times on EACH segment. This could cause performance degradation when each server is hosting lots of segments. In this PR, we introduced a new Object - ServerQuery to represent the query request on server side. It is already fully compiled and can be shared by all the segments. We can also add helper variables into this class to cache the result of the common operations for all segments to improve the performance. The new ServerQuery can also help isolating the execution layer from the transport layer. In the future if we decide to switch to another protocol for Broker-Server communication, we don't need to change anything for the execution layer. All we need is a new converter to convert the transport Object to ServerQuery. Minor fixes: - Support transform expression for IN and BETWEEN predicate This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] quietgopher commented on issue #5479: Vague error message during SegmentINdexCreation if column is not found
quietgopher commented on issue #5479: URL: https://github.com/apache/incubator-pinot/issues/5479#issuecomment-637213927 https://github.com/apache/incubator-pinot/pull/5482 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] quietgopher opened a new pull request #5482: Change a NPE to have a better error message
quietgopher opened a new pull request #5482: URL: https://github.com/apache/incubator-pinot/pull/5482 ## Description Improve an error message when a table spec is missing a dimension that is specified in a star tree schema. https://github.com/apache/incubator-pinot/issues/5479 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai opened a new pull request #5481: [TE] Fix the run-backend and run-frontend scripts
akshayrai opened a new pull request #5481: URL: https://github.com/apache/incubator-pinot/pull/5481 Updated the demo scripts to point to the right jar path. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5480: Adding more information into jar manifest
fx19880617 opened a new pull request #5480: URL: https://github.com/apache/incubator-pinot/pull/5480 ## Description Adding build time and branch into jar manifest Sample output from: `pinot-distribution/target/pinot-distribution-0.4.0-SNAPSHOT-shaded.jar:META-INF/MANIFEST.MF` ``` Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: xiangfu Build-Jdk: 1.8.0_232 Specification-Title: Pinot Distribution Specification-Version: 0.4.0-SNAPSHOT Specification-Vendor: Apache Software Foundation Build-Time: 2020-06-02T005003Z Implementation-Branch: git_plugin Implementation-Title: pinot-distribution Implementation-Vendor: Apache Software Foundation Implementation-Vendor-Id: org.apache.pinot Implementation-Version: 0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf235 75010cc76 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch git_plugin updated: Adding more information into jar manifest
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch git_plugin
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/git_plugin by this push:
new c96c73a Adding more information into jar manifest
c96c73a is described below
commit c96c73a6308f2f58bd0c6dcbe9eb6e43df9c2020
Author: Xiang Fu
AuthorDate: Mon Jun 1 17:59:22 2020 -0700
Adding more information into jar manifest
---
pom.xml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/pom.xml b/pom.xml
index 5b8c60e..c9b8d23 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1241,6 +1241,8 @@
+${maven.build.timestamp}
+${scmBranch}
${project.artifactId}
${project.version}-${buildNumber}
${project.groupId}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch git_plugin created (now ed26e85)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch git_plugin in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at ed26e85 [TE] fix document for adding new application (#5473) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] quietgopher opened a new issue #5479: Vague error message during SegmentINdexCreation if column is not found
quietgopher opened a new issue #5479: URL: https://github.com/apache/incubator-pinot/issues/5479 I haven't tracked down the issue with my schema. I'm guessing I have another typo. The logs look like this. I'm going to improve the error message. ``` java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:121) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:94) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:123) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] Caused by: java.lang.NullPointerException at org.apache.pinot.core.indexsegment.immutable.ImmutableSegmentImpl.getDictionary(ImmutableSegmentImpl.java:63) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.data.readers.PinotSegmentColumnReader.(PinotSegmentColumnReader.java:40) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.startree.v2.builder.BaseSingleTreeBuilder.(BaseSingleTreeBuilder.java:130) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.startree.v2.builder.OffHeapSingleTreeBuilder.(OffHeapSingleTreeBuilder.java:69) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder.getSingleTreeBuilder(MultipleTreesBuilder.java:140) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:118) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.buildStarTreeV2IfNecessary(SegmentIndexCreationDriverImpl.java:277) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:239) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:199) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.plugin.ingestion.batch.common.SegmentGenerationTaskRunner.run(SegmentGenerationTaskRunner.java:102) ~[pinot-batch-ingestion-standalone-0.4.0-SNAPSHOT-shaded.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.run(SegmentGenerationJobRunner.java:190) ~[pinot-batch-ingestion-standalone-0.4.0-SNAPSHOT-shaded.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:119) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] ... 4 more Exception caught: java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:121) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-ed26e8589fe5f91d2876d417aebf23575010cc76] at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:94) ~[pinot-all-0.4.0-SNAP
[GitHub] [incubator-pinot] chenboat opened a new pull request #5478: Add a new table config field for peer segment download.
chenboat opened a new pull request #5478: URL: https://github.com/apache/incubator-pinot/pull/5478 ## Description Add a new optional string field peerSegmentDownloadScheme to the SegmentsValidationAndRetentionConfig in the TableConfig. The value can be http or https. The field will enable download of segments for both realtime and offline table segments from peer servers. In the beginning, only realtime table segments download will be supported. The design details can be found [in this section ](https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+for+Realtime+segment+completion#By-passingdeep-storerequirementforRealtimesegmentcompletion-EnablebesteffortsegmentuploadinSplitSegmentCommiteranddownloadsegmentfrompeerservers.)of the cwiki doc. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) * [ ] Yes (Please label as **backward-incompat**, and complete the section below on Release Notes) Does this PR fix a zero-downtime upgrade introduced earlier? * [ ] Yes (Please label this as **backward-incompat**, and complete the section below on Release Notes) Does this PR otherwise need attention when creating release notes? Things to consider: - New configuration options - Deprecation of configurations - Signature changes to public methods/interfaces - New plugins added or old plugins removed * [ ] Yes (Please label this PR as **release-notes** and complete the section on Release Notes) ## Release Notes If you have tagged this as either backward-incompat or release-notes, you MUST add text here that you would like to see appear in release notes of the next release. If you have a series of commits adding or enabling a feature, then add this section only in final commit that marks the feature completed. Refer to earlier release notes to see examples of text ## Documentation If you have introduced a new feature or configuration, please add it to the documentation as well. See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5477: Upgrade pinot docker packaging and adding jmx agent
fx19880617 opened a new pull request #5477: URL: https://github.com/apache/incubator-pinot/pull/5477 ## Description Update docker packaging to be more intuitive. Adding jmx as etc to pinot docker image. ## Release Notes No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adding_jmx_to_pinot_docker_image created (now 77de993)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch adding_jmx_to_pinot_docker_image in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 77de993 Upgrade pinot docker packaging and adding jmx agent This branch includes the following new commits: new 77de993 Upgrade pinot docker packaging and adding jmx agent The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Upgrade pinot docker packaging and adding jmx agent
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch adding_jmx_to_pinot_docker_image
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 77de99331542c4147a5371bb6850934770d49cc4
Author: Xiang Fu
AuthorDate: Mon Jun 1 14:29:09 2020 -0700
Upgrade pinot docker packaging and adding jmx agent
---
docker/images/pinot/Dockerfile | 4 +-
.../etc/jmx_prometheus_javaagent/configs/pinot.yml | 208 +
.../jmx_prometheus_javaagent-0.12.0.jar| Bin 0 -> 370075 bytes
.../docker}/ingestion-job-specs/airlineStats.yaml | 0
.../docker}/ingestion-job-specs/baseballStats.yaml | 0
.../airlineStats_realtime_table_config.json| 0
.../meetupRsvp_realtime_table_config.json | 0
7 files changed, 210 insertions(+), 2 deletions(-)
diff --git a/docker/images/pinot/Dockerfile b/docker/images/pinot/Dockerfile
index 32c6a9b..e040697 100644
--- a/docker/images/pinot/Dockerfile
+++ b/docker/images/pinot/Dockerfile
@@ -72,9 +72,9 @@ RUN apt-get update && \
rm -rf /var/lib/apt/lists/*
COPY --from=pinot_build_env ${PINOT_HOME} ${PINOT_HOME}
-COPY ingestion-job-specs ${PINOT_HOME}/examples/docker/ingestion-job-specs
-COPY table-configs ${PINOT_HOME}/examples/docker/table-configs
COPY bin ${PINOT_HOME}/bin
+COPY etc ${PINOT_HOME}/etc
+COPY examples ${PINOT_HOME}/examples
# expose ports for controller/broker/server/admin
EXPOSE 9000 8099 8098 8097 8096
diff --git a/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
new file mode 100644
index 000..2f3d253
--- /dev/null
+++ b/docker/images/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml
@@ -0,0 +1,208 @@
+rules:
+# Pinot Controller
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_$1_$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_helix_$1_$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_helix_ZookeeperReconnects_$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_idealstateZnodeSize_$3"
+ labels:
+table: "$1"
+tableType: "$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_numberOfReplicas_$3"
+ labels:
+table: "$1"
+tableType: "$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_percentOfReplicas_$3"
+ labels:
+table: "$1"
+tableType: "$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_percentSegmentsAvailable_$3"
+ labels:
+table: "$1"
+tableType: "$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_segmentCount_$3"
+ labels:
+table: "$1"
+tableType: "$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_segmentsInErrorState_$3"
+ labels:
+table: "$1"
+tableType: "$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_numberSegmentUploadTimeoutExceeded_$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_numberTimesScheduleTasksCalled_$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_periodicTaskNumTablesProcessed_$1_$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_pinotControllerLeader_$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_partitionLeader_$1_$2"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_realtimeTableCount_$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_controller_validateion_$2_$3"
+ labels:
+table: "$1"
+# Pinot Broker
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_authorization_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_documentsScanned_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_entriesScannedInFilter_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_entriesScannedPostFilter_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_freshnessLagMs_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_queries_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_queryExecution_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.common.metrics\"<>(\\w+)"
+ name: "pinot_broker_queryRouting_$2"
+ labels:
+table: "$1"
+- pattern: "\"org.apache.pinot.comm
[incubator-pinot] branch master updated: [TE] fix document for adding new application (#5473)
This is an automated email from the ASF dual-hosted git repository. xhsun pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new ed26e85 [TE] fix document for adding new application (#5473) ed26e85 is described below commit ed26e8589fe5f91d2876d417aebf23575010cc76 Author: Xiaohui Sun AuthorDate: Mon Jun 1 14:11:46 2020 -0700 [TE] fix document for adding new application (#5473) Co-authored-by: Xiaohui Sun --- thirdeye/docs/configuration.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/thirdeye/docs/configuration.rst b/thirdeye/docs/configuration.rst index 7afe2fd..1115ee6 100644 --- a/thirdeye/docs/configuration.rst +++ b/thirdeye/docs/configuration.rst @@ -235,7 +235,7 @@ Example payload could be: "recipients": "owner-of-this-applicat...@company.com" } -2.From admin page +2.From admin page (only if there are existing applications) a. Navigate to admin page: http://localhost:1426/thirdeye-admin b. Select an existing application from "Entity Editor". - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] xiaohui-sun merged pull request #5473: [TE] fix document for adding new applications
xiaohui-sun merged pull request #5473: URL: https://github.com/apache/incubator-pinot/pull/5473 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5440: Add GenericTransformFunction wrapper for simple ScalarFunctions
KKcorps commented on a change in pull request #5440:
URL: https://github.com/apache/incubator-pinot/pull/5440#discussion_r433466964
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/TransformFunctionFactory.java
##
@@ -112,13 +115,24 @@ public static TransformFunction
get(TransformExpressionTree expression, Map transformFunctionClass =
TRANSFORM_FUNCTION_MAP.get(functionName);
+Class transformFunctionClass;
+FunctionInfo functionInfo = null;
+if (FunctionRegistry.containsFunctionByName(functionName)) {
+ transformFunctionClass = ScalarTransformFunctionWrapper.class;
+ functionInfo = FunctionRegistry.getFunctionByName(functionName);
+} else {
+ transformFunctionClass = TRANSFORM_FUNCTION_MAP.get(functionName);
+}
+
if (transformFunctionClass == null) {
throw new BadQueryRequestException("Unsupported transform function:
" + functionName);
}
try {
transformFunction = transformFunctionClass.newInstance();
-} catch (InstantiationException | IllegalAccessException e) {
+ if (functionInfo != null) {
+((ScalarTransformFunctionWrapper)
transformFunction).setFunction(functionName, functionInfo);
+ }
+} catch (Exception e) {
Review comment:
Why will a constructor throw BadQueryRequestException since this catch
block is meant for only the constructor?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5440: Add GenericTransformFunction wrapper for simple ScalarFunctions
KKcorps commented on a change in pull request #5440:
URL: https://github.com/apache/incubator-pinot/pull/5440#discussion_r433462644
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java
##
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.FunctionInfo;
+import org.apache.pinot.common.function.FunctionInvoker;
+import org.apache.pinot.core.common.DataSource;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class ScalarTransformFunctionWrapper extends BaseTransformFunction {
Review comment:
IMO, this is a wrapper/adapater around ScalarTransformFunctions hence
the name
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5440: Add GenericTransformFunction wrapper for simple ScalarFunctions
KKcorps commented on a change in pull request #5440:
URL: https://github.com/apache/incubator-pinot/pull/5440#discussion_r433450882
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java
##
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.FunctionInfo;
+import org.apache.pinot.common.function.FunctionInvoker;
+import org.apache.pinot.core.common.DataSource;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class ScalarTransformFunctionWrapper extends BaseTransformFunction {
+
+ FunctionInvoker _functionInvoker;
+ String _name;
+ Object[] _args;
+ List _nonLiteralArgIndices;
+ List _nonLiteralArgType;
+ List _nonLiteralTransformFunction;
+ TransformResultMetadata _transformResultMetadata;
+ String[] _stringResult;
+ int[] _integerResult;
+ float[] _floatResult;
+ double[] _doubleResult;
+ long[] _longResult;
+
+ public ScalarTransformFunctionWrapper() {
+_nonLiteralArgIndices = new ArrayList<>();
+_nonLiteralArgType = new ArrayList<>();
+_nonLiteralTransformFunction = new ArrayList<>();
+ }
+
+ @Override
+ public String getName() {
+return _name;
+ }
+
+ public void setFunction(String functionName, FunctionInfo info)
+ throws Exception {
+_name = functionName;
+_functionInvoker = new FunctionInvoker(info);
+ }
+
+ @Override
+ public void init(List arguments, Map
dataSourceMap) {
+Preconditions.checkArgument(arguments.size() ==
_functionInvoker.getParameterTypes().length,
+"The number of arguments are not same for scalar function and
transform function: %s", getName());
+
+_args = new Object[arguments.size()];
+for (int i = 0; i < arguments.size(); i++) {
+ TransformFunction function = arguments.get(i);
+ if (function instanceof LiteralTransformFunction) {
+String literal = ((LiteralTransformFunction) function).getLiteral();
+Class paramType = _functionInvoker.getParameterTypes()[i];
Review comment:
It would require changing switch case to a lot of else if since switch
doesn't accept Class type
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5440: Add GenericTransformFunction wrapper for simple ScalarFunctions
KKcorps commented on a change in pull request #5440:
URL: https://github.com/apache/incubator-pinot/pull/5440#discussion_r433450882
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function/ScalarTransformFunctionWrapper.java
##
@@ -0,0 +1,301 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform.function;
+
+import com.google.common.base.Preconditions;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.function.FunctionInfo;
+import org.apache.pinot.common.function.FunctionInvoker;
+import org.apache.pinot.core.common.DataSource;
+import org.apache.pinot.core.operator.blocks.ProjectionBlock;
+import org.apache.pinot.core.operator.transform.TransformResultMetadata;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class ScalarTransformFunctionWrapper extends BaseTransformFunction {
+
+ FunctionInvoker _functionInvoker;
+ String _name;
+ Object[] _args;
+ List _nonLiteralArgIndices;
+ List _nonLiteralArgType;
+ List _nonLiteralTransformFunction;
+ TransformResultMetadata _transformResultMetadata;
+ String[] _stringResult;
+ int[] _integerResult;
+ float[] _floatResult;
+ double[] _doubleResult;
+ long[] _longResult;
+
+ public ScalarTransformFunctionWrapper() {
+_nonLiteralArgIndices = new ArrayList<>();
+_nonLiteralArgType = new ArrayList<>();
+_nonLiteralTransformFunction = new ArrayList<>();
+ }
+
+ @Override
+ public String getName() {
+return _name;
+ }
+
+ public void setFunction(String functionName, FunctionInfo info)
+ throws Exception {
+_name = functionName;
+_functionInvoker = new FunctionInvoker(info);
+ }
+
+ @Override
+ public void init(List arguments, Map
dataSourceMap) {
+Preconditions.checkArgument(arguments.size() ==
_functionInvoker.getParameterTypes().length,
+"The number of arguments are not same for scalar function and
transform function: %s", getName());
+
+_args = new Object[arguments.size()];
+for (int i = 0; i < arguments.size(); i++) {
+ TransformFunction function = arguments.get(i);
+ if (function instanceof LiteralTransformFunction) {
+String literal = ((LiteralTransformFunction) function).getLiteral();
+Class paramType = _functionInvoker.getParameterTypes()[i];
Review comment:
It would require changing switch case to a lot of else if since switch
doesn't accept Class type. IMO, switch case looks neat so made this trade off
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: bring pinot mock data generator.sh script in line with TE mock data generator
This is an automated email from the ASF dual-hosted git repository.
apucher pushed a commit to branch mock-data-generator-range-extension
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 8cde6b9738284b6d19e111322dd9cf0a99f22735
Author: Alexander Pucher
AuthorDate: Mon Jun 1 11:22:05 2020 -0700
bring pinot mock data generator.sh script in line with TE mock data
generator
---
docker/images/pinot/bin/generator.sh | 2 +-
pinot-tools/src/main/resources/generator/README.md | 8 ++-
.../generator/complexWebsite_generator.json| 2 +-
.../src/main/resources/generator/generator.sh | 76 --
.../generator/simpleWebsite_generator.json | 2 +-
5 files changed, 9 insertions(+), 81 deletions(-)
diff --git a/docker/images/pinot/bin/generator.sh
b/docker/images/pinot/bin/generator.sh
index dc84967..a803f30 100755
--- a/docker/images/pinot/bin/generator.sh
+++ b/docker/images/pinot/bin/generator.sh
@@ -42,7 +42,7 @@ echo "Extracting template files"
echo "Generating data for ${TEMPLATE_NAME} in ${DATA_DIR}"
${ADMIN_PATH} GenerateData \
--numFiles 1 -numRecords 354780 -format csv \
+-numFiles 1 -numRecords 946080 -format csv \
-schemaFile "${TEMPLATE_BASEDIR}/${TEMPLATE_NAME}_schema.json" \
-schemaAnnotationFile "${TEMPLATE_BASEDIR}/${TEMPLATE_NAME}_generator.json" \
-outDir "$DATA_DIR"
diff --git a/pinot-tools/src/main/resources/generator/README.md
b/pinot-tools/src/main/resources/generator/README.md
index f2a3b89..0123cc7 100644
--- a/pinot-tools/src/main/resources/generator/README.md
+++ b/pinot-tools/src/main/resources/generator/README.md
@@ -31,6 +31,9 @@ data with these patterns if you so desire.
The command line examples below are meant to be executed from the **pinot
repository root**.
(This was tested with pinot-quickstart in batch mode. Requires DefaultTenant
and broker)
+NOTE: there's a `generator.sh` script under
`incubator-pinot/docker/images/pinot/bin` that wraps this
+functionality for Apache Pinot docker containers.
+
## Generate data via pattern
This first step generates the raw data from a given generator file. By
default, we generate the data as CSV, and you can
have a look manually with your favorite spreadsheet tool.
@@ -39,7 +42,7 @@ have a look manually with your favorite spreadsheet tool.
```
./pinot-tools/target/pinot-tools-pkg/bin/pinot-admin.sh GenerateData \
--numFiles 1 -numRecords 354780 -format csv \
+-numFiles 1 -numRecords 946080 -format csv \
-schemaFile
./pinot-tools/src/main/resources/generator/complexWebsite_schema.json \
-schemaAnnotationFile
./pinot-tools/src/main/resources/generator/complexWebsite_generator.json \
-outDir ./myTestData
@@ -50,7 +53,8 @@ Now we turn the verbose CSV data into an efficiently packed
segment ready for up
```
./pinot-tools/target/pinot-tools-pkg/bin/pinot-admin.sh CreateSegment \
--tableName complexWebsite -segmentName complexWebsite -format CSV -overwrite \
+-format CSV \
+-tableConfigFile
./pinot-tools/src/main/resources/generator/complexWebsite_config.json \
-schemaFile
./pinot-tools/src/main/resources/generator/complexWebsite_schema.json \
-dataDir ./myTestData \
-outDir ./myTestSegment
diff --git
a/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
b/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
index 41f08c9..a2ee301 100644
--- a/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
+++ b/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
@@ -2,7 +2,7 @@
{
"column": "hoursSinceEpoch",
"pattern": {
- "type": "SEQUENCE", "start": 420768, "stepsize": 1, "repetitions": 18
+ "type": "SEQUENCE", "start": 429528, "stepsize": 1, "repetitions": 18
}
},
{
diff --git a/pinot-tools/src/main/resources/generator/generator.sh
b/pinot-tools/src/main/resources/generator/generator.sh
deleted file mode 100755
index ac1e746..000
--- a/pinot-tools/src/main/resources/generator/generator.sh
+++ /dev/null
@@ -1,76 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-
-#ADMIN_PATH="/opt/pinot/bin/pinot-admin.sh"
-ADMIN_PATH="./pinot-tools/target/pinot-tool
[incubator-pinot] branch mock-data-generator-range-extension created (now 8cde6b9)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch mock-data-generator-range-extension in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 8cde6b9 bring pinot mock data generator.sh script in line with TE mock data generator This branch includes the following new commits: new 8cde6b9 bring pinot mock data generator.sh script in line with TE mock data generator The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch mock-data-generator-range-extension created (now 8cde6b9)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch mock-data-generator-range-extension in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 8cde6b9 bring pinot mock data generator.sh script in line with TE mock data generator This branch includes the following new commits: new 8cde6b9 bring pinot mock data generator.sh script in line with TE mock data generator The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: bring pinot mock data generator.sh script in line with TE mock data generator
This is an automated email from the ASF dual-hosted git repository.
apucher pushed a commit to branch mock-data-generator-range-extension
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 8cde6b9738284b6d19e111322dd9cf0a99f22735
Author: Alexander Pucher
AuthorDate: Mon Jun 1 11:22:05 2020 -0700
bring pinot mock data generator.sh script in line with TE mock data
generator
---
docker/images/pinot/bin/generator.sh | 2 +-
pinot-tools/src/main/resources/generator/README.md | 8 ++-
.../generator/complexWebsite_generator.json| 2 +-
.../src/main/resources/generator/generator.sh | 76 --
.../generator/simpleWebsite_generator.json | 2 +-
5 files changed, 9 insertions(+), 81 deletions(-)
diff --git a/docker/images/pinot/bin/generator.sh
b/docker/images/pinot/bin/generator.sh
index dc84967..a803f30 100755
--- a/docker/images/pinot/bin/generator.sh
+++ b/docker/images/pinot/bin/generator.sh
@@ -42,7 +42,7 @@ echo "Extracting template files"
echo "Generating data for ${TEMPLATE_NAME} in ${DATA_DIR}"
${ADMIN_PATH} GenerateData \
--numFiles 1 -numRecords 354780 -format csv \
+-numFiles 1 -numRecords 946080 -format csv \
-schemaFile "${TEMPLATE_BASEDIR}/${TEMPLATE_NAME}_schema.json" \
-schemaAnnotationFile "${TEMPLATE_BASEDIR}/${TEMPLATE_NAME}_generator.json" \
-outDir "$DATA_DIR"
diff --git a/pinot-tools/src/main/resources/generator/README.md
b/pinot-tools/src/main/resources/generator/README.md
index f2a3b89..0123cc7 100644
--- a/pinot-tools/src/main/resources/generator/README.md
+++ b/pinot-tools/src/main/resources/generator/README.md
@@ -31,6 +31,9 @@ data with these patterns if you so desire.
The command line examples below are meant to be executed from the **pinot
repository root**.
(This was tested with pinot-quickstart in batch mode. Requires DefaultTenant
and broker)
+NOTE: there's a `generator.sh` script under
`incubator-pinot/docker/images/pinot/bin` that wraps this
+functionality for Apache Pinot docker containers.
+
## Generate data via pattern
This first step generates the raw data from a given generator file. By
default, we generate the data as CSV, and you can
have a look manually with your favorite spreadsheet tool.
@@ -39,7 +42,7 @@ have a look manually with your favorite spreadsheet tool.
```
./pinot-tools/target/pinot-tools-pkg/bin/pinot-admin.sh GenerateData \
--numFiles 1 -numRecords 354780 -format csv \
+-numFiles 1 -numRecords 946080 -format csv \
-schemaFile
./pinot-tools/src/main/resources/generator/complexWebsite_schema.json \
-schemaAnnotationFile
./pinot-tools/src/main/resources/generator/complexWebsite_generator.json \
-outDir ./myTestData
@@ -50,7 +53,8 @@ Now we turn the verbose CSV data into an efficiently packed
segment ready for up
```
./pinot-tools/target/pinot-tools-pkg/bin/pinot-admin.sh CreateSegment \
--tableName complexWebsite -segmentName complexWebsite -format CSV -overwrite \
+-format CSV \
+-tableConfigFile
./pinot-tools/src/main/resources/generator/complexWebsite_config.json \
-schemaFile
./pinot-tools/src/main/resources/generator/complexWebsite_schema.json \
-dataDir ./myTestData \
-outDir ./myTestSegment
diff --git
a/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
b/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
index 41f08c9..a2ee301 100644
--- a/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
+++ b/pinot-tools/src/main/resources/generator/complexWebsite_generator.json
@@ -2,7 +2,7 @@
{
"column": "hoursSinceEpoch",
"pattern": {
- "type": "SEQUENCE", "start": 420768, "stepsize": 1, "repetitions": 18
+ "type": "SEQUENCE", "start": 429528, "stepsize": 1, "repetitions": 18
}
},
{
diff --git a/pinot-tools/src/main/resources/generator/generator.sh
b/pinot-tools/src/main/resources/generator/generator.sh
deleted file mode 100755
index ac1e746..000
--- a/pinot-tools/src/main/resources/generator/generator.sh
+++ /dev/null
@@ -1,76 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-#
-
-
-#ADMIN_PATH="/opt/pinot/bin/pinot-admin.sh"
-ADMIN_PATH="./pinot-tools/target/pinot-tool
[GitHub] [incubator-pinot] quietgopher commented on issue #5474: Vague error message when building startree schema if there is a typo in the aggregate columns
quietgopher commented on issue #5474: URL: https://github.com/apache/incubator-pinot/issues/5474#issuecomment-636970008 https://github.com/apache/incubator-pinot/pull/5476 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] quietgopher opened a new pull request #5476: Improve error message on NPE related to typo in star schema column names
quietgopher opened a new pull request #5476: URL: https://github.com/apache/incubator-pinot/pull/5476 ## Description I hit an NPE due to a typo in my star tree schema. This diff includes the invalid column name in the NPE now (to make it easier to debug). https://github.com/apache/incubator-pinot/issues/5474 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] quietgopher commented on issue #5474: Vague error message when building startree schema if there is a typo in the aggregate columns
quietgopher commented on issue #5474:
URL:
https://github.com/apache/incubator-pinot/issues/5474#issuecomment-636968434
I'm working on a diff.
Here's an example of the copy/paste issue (simplified since I don't want to
post the full config).
in the table config:
```
{
"tableName": "metrics",
...
"tableIndexConfig" : {
"starTreeIndexConfigs": [
{
...
"functionColumnPairs": [
"SUM__impressions",
...
]
},
```
then in the metrics schema:
```
{
"schemaName": "metrics",
"metricFieldSpecs": [
...
{
"name": "SUM__impressions",
"dataType": "LONG"
},
...
]
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Sreemanth opened a new issue #5475: GuavaOnHeapBloomFilter can be constructed with _bloomFilter as null
Sreemanth opened a new issue #5475: URL: https://github.com/apache/incubator-pinot/issues/5475 https://github.com/apache/incubator-pinot/blob/a314d42e8744549e5c182383445f11c60ac4ae4a/pinot-core/src/main/java/org/apache/pinot/core/bloom/GuavaOnHeapBloomFilter.java#L37 Because of default constructor _bloomFilter becomes null. `add` method of `GuavaOnHeapBloomFilter` will throw `NullPointerException` error. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5474: Vague error message when building startree schema if there is a typo in the aggregate columns
kishoreg commented on issue #5474: URL: https://github.com/apache/incubator-pinot/issues/5474#issuecomment-636669663 can you paste the sample config This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
