[incubator-pinot] branch configurable_spark_job_parallelism updated (6e2cf59 -> 35b7ebc)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch configurable_spark_job_parallelism in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 6e2cf59 Adding field 'segmentCreationJobParallelism' to allow users to set segment generation job parallelism. Default to the number of input files. add 35b7ebc Adding field 'segmentCreationJobParallelism' to allow users to set segment generation job parallelism. Default to the number of input files. This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (6e2cf59) \ N -- N -- N refs/heads/configurable_spark_job_parallelism (35b7ebc) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java | 7 --- .../ingestion/batch/spark/SparkSegmentGenerationJobRunner.java | 7 +-- 2 files changed, 9 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #6012: Configurable segment generation job parallelism for Hadoop and Spark
fx19880617 commented on a change in pull request #6012: URL: https://github.com/apache/incubator-pinot/pull/6012#discussion_r488431110 ## File path: pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/resources/segmentCreationAndTarPushJobSpec.yaml ## @@ -28,6 +28,7 @@ includeFileNamePattern: 'glob:**/*.parquet' excludeFileNamePattern: 'glob:**/*.avro' outputDirURI: 'file:///path/to/output' overwriteOutput: true +parallelism: 1 Review comment: it means 10k tasks in Spark or mappers in Hadoop ;) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #6012: Configurable segment generation job parallelism for Hadoop and Spark
fx19880617 commented on a change in pull request #6012:
URL: https://github.com/apache/incubator-pinot/pull/6012#discussion_r488432050
##
File path:
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
##
@@ -230,7 +230,10 @@ public void run()
if (hadoopTokenFileLocation != null) {
jobConf.set("mapreduce.job.credentials.binary",
hadoopTokenFileLocation);
}
- jobConf.setInt(JobContext.NUM_MAPS, numDataFiles);
+
+ int jobParallelism =
+ (_spec.getSegmentCreationJobParallelism() > 0) ?
_spec.getSegmentCreationJobParallelism() : numDataFiles;
Review comment:
You are right :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] shahsank3t commented on pull request #5981: Support for Update & Delete in ZooKeeper Browser and added SQL Functions in SQL Editor autocomplete list
shahsank3t commented on pull request #5981: URL: https://github.com/apache/incubator-pinot/pull/5981#issuecomment-692519826 @kishoreg rebased and all checks passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Adding field 'segmentCreationJobParallelism' to allow users to set segment generation job parallelism. Default to the number of input files. (#6012)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 054faf7 Adding field 'segmentCreationJobParallelism' to allow users
to set segment generation job parallelism. Default to the number of input
files. (#6012)
054faf7 is described below
commit 054faf76cdef5a625d042a4d435d428529c8d798
Author: Xiang Fu
AuthorDate: Tue Sep 15 02:20:12 2020 -0700
Adding field 'segmentCreationJobParallelism' to allow users to set segment
generation job parallelism. Default to the number of input files. (#6012)
---
.../batch/hadoop/HadoopSegmentGenerationJobRunner.java | 6 +-
.../batch/spark/SparkSegmentGenerationJobRunner.java| 12 +---
.../main/resources/segmentCreationAndTarPushJobSpec.yaml| 1 +
.../spi/ingestion/batch/spec/SegmentGenerationJobSpec.java | 13 +
.../pinot/spi/ingestion/batch/IngestionJobLauncherTest.java | 2 ++
.../src/test/resources/ingestion_job_spec_template.yaml | 1 +
6 files changed, 31 insertions(+), 4 deletions(-)
diff --git
a/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
b/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
index dee13e5..59beadc 100644
---
a/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
+++
b/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
@@ -230,7 +230,11 @@ public class HadoopSegmentGenerationJobRunner extends
Configured implements Inge
if (hadoopTokenFileLocation != null) {
jobConf.set("mapreduce.job.credentials.binary",
hadoopTokenFileLocation);
}
- jobConf.setInt(JobContext.NUM_MAPS, numDataFiles);
+ int jobParallelism = _spec.getSegmentCreationJobParallelism();
+ if (jobParallelism <= 0 || jobParallelism > numDataFiles) {
+jobParallelism = numDataFiles;
+ }
+ jobConf.setInt(JobContext.NUM_MAPS, jobParallelism);
// Pinot plugins are necessary to launch Pinot ingestion job from every
mapper.
// In order to ensure pinot plugins would be loaded to each worker, this
method
diff --git
a/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java
b/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java
index c1b3f25..cdee5cf 100644
---
a/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java
+++
b/pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-spark/src/main/java/org/apache/pinot/plugin/ingestion/batch/spark/SparkSegmentGenerationJobRunner.java
@@ -205,7 +205,8 @@ public class SparkSegmentGenerationJobRunner implements
IngestionJobRunner, Seri
}
List pathAndIdxList = new ArrayList<>();
- String localDirectorySequenceIdString =
_spec.getSegmentNameGeneratorSpec().getConfigs().get(LOCAL_DIRECTORY_SEQUENCE_ID);
+ String localDirectorySequenceIdString =
+
_spec.getSegmentNameGeneratorSpec().getConfigs().get(LOCAL_DIRECTORY_SEQUENCE_ID);
boolean localDirectorySequenceId = false;
if (localDirectorySequenceIdString != null) {
localDirectorySequenceId =
Boolean.parseBoolean(localDirectorySequenceIdString);
@@ -219,7 +220,7 @@ public class SparkSegmentGenerationJobRunner implements
IngestionJobRunner, Seri
}
localDirIndex.get(filteredParentPath.toString()).add(filteredFile);
}
-for (String parentPath: localDirIndex.keySet()){
+for (String parentPath : localDirIndex.keySet()) {
List siblingFiles = localDirIndex.get(parentPath);
Collections.sort(siblingFiles);
for (int i = 0; i < siblingFiles.size(); i++) {
@@ -231,7 +232,12 @@ public class SparkSegmentGenerationJobRunner implements
IngestionJobRunner, Seri
pathAndIdxList.add(String.format("%s %d", filteredFiles.get(i), i));
}
}
- JavaRDD pathRDD = sparkContext.parallelize(pathAndIdxList,
pathAndIdxList.size());
+ int numDataFiles = pathAndIdxList.size();
+ int jobParallelism = _spec.getSegmentCreationJobParallelism();
+ if (jobParallelism <= 0 || jobParallelism > numDataFiles) {
+jobParallelism = numDataFiles;
+ }
+
[GitHub] [incubator-pinot] fx19880617 merged pull request #6012: Configurable segment generation job parallelism for Hadoop and Spark
fx19880617 merged pull request #6012: URL: https://github.com/apache/incubator-pinot/pull/6012 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Support for Update & Delete in ZooKeeper Browser and added SQL Functions in SQL Editor autocomplete list (#5981)
This is an automated email from the ASF dual-hosted git repository.
kishoreg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 5da3433 Support for Update & Delete in ZooKeeper Browser and added
SQL Functions in SQL Editor autocomplete list (#5981)
5da3433 is described below
commit 5da34330bce20ea1f4b86b47b2b2266a764e29c1
Author: Sanket Shah
AuthorDate: Tue Sep 15 19:38:16 2020 +0530
Support for Update & Delete in ZooKeeper Browser and added SQL Functions in
SQL Editor autocomplete list (#5981)
* Adding api to edit ZK path
* Adding delete api
* Support for Update & Delete in ZooKeeper Browser and added SQL Functions
in SQL Editor autocomplete list
* showing notification on operation completion, display last refresh time,
fixed refresh action
Co-authored-by: kishoreg
---
.../src/main/resources/app/components/Confirm.tsx | 106
.../resources/app/components/CustomCodemirror.tsx | 66 ++
.../app/components/Zookeeper/TreeDirectory.tsx | 136 +++--
.../src/main/resources/app/interfaces/types.d.ts | 1 +
.../src/main/resources/app/pages/Query.tsx | 9 ++
.../src/main/resources/app/pages/ZookeeperPage.tsx | 63 +-
.../src/main/resources/app/requests/index.ts | 10 +-
.../src/main/resources/app/styles/styles.css | 5 +
.../main/resources/app/utils/PinotMethodUtils.ts | 22 +++-
.../src/main/resources/app/utils/Utils.tsx | 31 ++---
.../src/main/resources/app/utils/axios-config.ts | 2 +-
11 files changed, 394 insertions(+), 57 deletions(-)
diff --git a/pinot-controller/src/main/resources/app/components/Confirm.tsx
b/pinot-controller/src/main/resources/app/components/Confirm.tsx
new file mode 100644
index 000..bb11f99
--- /dev/null
+++ b/pinot-controller/src/main/resources/app/components/Confirm.tsx
@@ -0,0 +1,106 @@
+/* eslint-disable no-nested-ternary */
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import React, { useEffect } from 'react';
+import { Button, Dialog, DialogTitle, DialogContent, DialogContentText,
DialogActions, makeStyles } from '@material-ui/core';
+import { green, red } from '@material-ui/core/colors';
+
+const useStyles = makeStyles((theme) => ({
+ dialogContent: {
+minWidth: 900
+ },
+ dialogTextContent: {
+fontWeight: 600
+ },
+ dialogActions: {
+justifyContent: 'center'
+ },
+ green: {
+fontWeight: 600,
+color: green[500],
+borderColor: green[500],
+'&:hover': {
+ backgroundColor: green[50],
+ borderColor: green[500]
+}
+ },
+ red: {
+fontWeight: 600,
+color: red[500],
+borderColor: red[500],
+'&:hover': {
+ backgroundColor: red[50],
+ borderColor: red[500]
+}
+ }
+}));
+
+
+type Props = {
+ openDialog: boolean,
+ dialogTitle?: string,
+ dialogContent: string,
+ successCallback: (event: React.MouseEvent) =>
void;
+ closeDialog: (event: React.MouseEvent) =>
void;
+ dialogYesLabel?: string,
+ dialogNoLabel?: string
+};
+
+const Confirm = ({openDialog, dialogTitle, dialogContent, successCallback,
closeDialog, dialogYesLabel, dialogNoLabel}: Props) => {
+ const classes = useStyles();
+ const [open, setOpen] = React.useState(openDialog);
+
+ useEffect(()=>{
+setOpen(openDialog);
+ }, [openDialog])
+
+ const isStringDialog = typeof dialogContent === 'string';
+
+ return (
+
+
+{dialogTitle && {dialogTitle}}
+
+ {isStringDialog ?
+
+ {dialogContent}
+
+ : dialogContent}
+
+
+
+{dialogNoLabel || "No"}
+
+
+{dialogYesLabel || "Yes"}
+
+
+
+
+ );
+};
+
+export default Confirm;
\ No newline at end of file
diff --git
a/pinot-controller/src/main/resources/app/components/CustomCodemirror.tsx
b/pinot-controller/src/main/resources/app/components/CustomCodemirror.tsx
new file mode 100644
index 000..4e55dec
--- /dev/null
+++
[GitHub] [incubator-pinot] kishoreg merged pull request #5981: Support for Update & Delete in ZooKeeper Browser and added SQL Functions in SQL Editor autocomplete list
kishoreg merged pull request #5981: URL: https://github.com/apache/incubator-pinot/pull/5981 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] lgo edited a comment on issue #5814: Pinot's swagger pages does not work on HTTPS-only environments
lgo edited a comment on issue #5814: URL: https://github.com/apache/incubator-pinot/issues/5814#issuecomment-692855728 This has come up against in a different shape after Swagger was bumpted to v3 (#5888). This time, the page does list `HTTP` and `HTTPS` because the Swagger resources list those schemas, but the page does not correctly select the currently used resources which the previous version of Swagger did. Instead it selects the first schema in the list of schemas provided in the resources. This results in any request having the user error ``` TypeError: Failed to fetch ``` While failing due to making an HTTP request, as seen in the logs ``` Mixed Content: The page at 'https://controller-host/help#/Schema/addSchema' was loaded over HTTPS, but requested an insecure resource 'http://controller-host/schemas?override=true'. This request has been blocked; the content must be served over HTTPS. (anonymous) @ index.js:1 ``` https://user-images.githubusercontent.com/1161183/93242979-6999b180-f73c-11ea-8b2f-469dc74c6919.png;> https://user-images.githubusercontent.com/1161183/93242882-4a028900-f73c-11ea-8e7a-e3a1093c9f92.png;> I reckon this might be a similar issue as before and there may be a Swagger option to have it automatically select HTTP vs HTTPS in Swagger UI / Swagger JS. It is posible to select the HTTPS resource, so this is mostly a usability / conveneience issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6017: Table indexing config validation
jackjlli commented on a change in pull request #6017:
URL: https://github.com/apache/incubator-pinot/pull/6017#discussion_r488818486
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##
@@ -65,6 +68,7 @@ public static void validate(TableConfig tableConfig,
@Nullable Schema schema) {
validateValidationConfig(tableConfig, schema);
validateIngestionConfig(tableConfig.getIngestionConfig(), schema);
validateTierConfigList(tableConfig.getTierConfigsList());
+validateIndexingConfig(tableConfig.getIndexingConfig(), schema);
Review comment:
Could you update the description of the method `validate` as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] lgo commented on issue #5814: Pinot's swagger pages does not work on HTTPS-only environments
lgo commented on issue #5814: URL: https://github.com/apache/incubator-pinot/issues/5814#issuecomment-692855728 This has come up against in a different shape after Swagger was bumpted to v3 (#5888). This time, the page does list `HTTP` and `HTTPS` because the Swagger resources list those schemas, but the page does not correctly select the currently used resources which the previous version of Swagger did. This results in any request having the user error ``` TypeError: Failed to fetch ``` While failing due to making an HTTP request, as seen in the logs ``` Mixed Content: The page at 'https://controller-host/help#/Schema/addSchema' was loaded over HTTPS, but requested an insecure resource 'http://controller-host/schemas?override=true'. This request has been blocked; the content must be served over HTTPS. (anonymous) @ index.js:1 ``` https://user-images.githubusercontent.com/1161183/93242979-6999b180-f73c-11ea-8b2f-469dc74c6919.png;> https://user-images.githubusercontent.com/1161183/93242882-4a028900-f73c-11ea-8e7a-e3a1093c9f92.png;> I reckon this might be a similar issue as before and there may be a Swagger option to have it automatically select HTTP vs HTTPS in Swagger UI / Swagger JS. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] lgo edited a comment on issue #5814: Pinot's swagger pages does not work on HTTPS-only environments
lgo edited a comment on issue #5814: URL: https://github.com/apache/incubator-pinot/issues/5814#issuecomment-692855728 This has come up against in a different shape after Swagger was bumpted to v3 (#5888). This time, the page does list `HTTP` and `HTTPS` because the Swagger resources list those schemas, but the page does not correctly select the currently used resources which the previous version of Swagger did. Instead it selects the first schema in the list of schemas provided in the resources. This results in any request having the user error ``` TypeError: Failed to fetch ``` While failing due to making an HTTP request, as seen in the logs ``` Mixed Content: The page at 'https://controller-host/help#/Schema/addSchema' was loaded over HTTPS, but requested an insecure resource 'http://controller-host/schemas?override=true'. This request has been blocked; the content must be served over HTTPS. (anonymous) @ index.js:1 ``` https://user-images.githubusercontent.com/1161183/93242979-6999b180-f73c-11ea-8b2f-469dc74c6919.png;> https://user-images.githubusercontent.com/1161183/93242882-4a028900-f73c-11ea-8e7a-e3a1093c9f92.png;> I reckon this might be a similar issue as before and there may be a Swagger option to have it automatically select HTTP vs HTTPS in Swagger UI / Swagger JS. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #5934: Segment processing framework
npawar commented on a change in pull request #5934:
URL: https://github.com/apache/incubator-pinot/pull/5934#discussion_r488812749
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/utils/SegmentProcessorUtils.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.processing.utils;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.generic.GenericData;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * Helper util methods for SegmentProcessorFramework
+ */
+public final class SegmentProcessorUtils {
+
+ private SegmentProcessorUtils() {
+ }
+
+ /**
+ * Convert a GenericRow to an avro GenericRecord
+ */
+ public static GenericData.Record convertGenericRowToAvroRecord(GenericRow
genericRow,
+ GenericData.Record reusableRecord) {
+for (String field : genericRow.getFieldToValueMap().keySet()) {
Review comment:
I see, got it. In the interest of getting this long standing and big PR
merged, I will think about this and take it up immediately in a following PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6017: Table indexing config validation
mcvsubbu commented on a change in pull request #6017:
URL: https://github.com/apache/incubator-pinot/pull/6017#discussion_r488791747
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##
@@ -225,4 +229,54 @@ private static void validateTierConfigList(@Nullable
List tierConfig
}
}
}
+
+ private static void validateIndexingConfig(@Nullable IndexingConfig
indexingConfig, @Nullable Schema schema) {
+if (indexingConfig == null || schema == null) {
Review comment:
Can you also add FieldConfig validation here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #5934: Segment processing framework
npawar commented on a change in pull request #5934:
URL: https://github.com/apache/incubator-pinot/pull/5934#discussion_r488812122
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/framework/SegmentMapper.java
##
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.processing.framework;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericDatumWriter;
+import org.apache.pinot.core.data.readers.PinotSegmentRecordReader;
+import org.apache.pinot.core.segment.processing.partitioner.PartitionFilter;
+import org.apache.pinot.core.segment.processing.partitioner.Partitioner;
+import org.apache.pinot.core.segment.processing.partitioner.PartitionerFactory;
+import org.apache.pinot.core.segment.processing.transformer.RecordTransformer;
+import
org.apache.pinot.core.segment.processing.transformer.RecordTransformerFactory;
+import org.apache.pinot.core.segment.processing.utils.SegmentProcessorUtils;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Mapper phase of the SegmentProcessorFramework.
+ * Reads the input segment and creates partitioned avro data files
+ * Performs:
+ * - record transformations
+ * - partitioning
+ * - partition filtering
+ */
+public class SegmentMapper {
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(SegmentMapper.class);
+ private final File _inputSegment;
+ private final File _mapperOutputDir;
+
+ private final String _mapperId;
+ private final Schema _avroSchema;
+ private final RecordTransformer _recordTransformer;
+ private final Partitioner _partitioner;
+ private final PartitionFilter _partitionFilter;
+ private final Map>
_partitionToDataFileWriterMap = new HashMap<>();
+
+ public SegmentMapper(String mapperId, File inputSegment, SegmentMapperConfig
mapperConfig, File mapperOutputDir) {
+_inputSegment = inputSegment;
+_mapperOutputDir = mapperOutputDir;
+
+_mapperId = mapperId;
+_avroSchema =
SegmentProcessorUtils.convertPinotSchemaToAvroSchema(mapperConfig.getPinotSchema());
+_recordTransformer =
RecordTransformerFactory.getRecordTransformer(mapperConfig.getRecordTransformerConfig());
+_partitioner =
PartitionerFactory.getPartitioner(mapperConfig.getPartitioningConfig());
+_partitionFilter =
PartitionerFactory.getPartitionFilter(mapperConfig.getPartitioningConfig());
+LOGGER.info(
+"Initialized mapper with id: {}, input segment: {}, output dir: {},
recordTransformer: {}, partitioner: {}, partitionFilter: {}",
+_mapperId, _inputSegment, _mapperOutputDir,
_recordTransformer.getClass(), _partitioner.getClass(),
+_partitionFilter.getClass());
+ }
+
+ /**
+ * Reads the input segment and generates partitioned avro data files into
the mapper output directory
+ * Records for each partition are put into a directory of its own withing
the mapper output directory, identified by the partition name
+ */
+ public void map()
+ throws Exception {
+
+PinotSegmentRecordReader segmentRecordReader = new
PinotSegmentRecordReader(_inputSegment);
+GenericRow reusableRow = new GenericRow();
+GenericData.Record reusableRecord = new GenericData.Record(_avroSchema);
Review comment:
Added TODO
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (5da3433 -> 41de9a6)
This is an automated email from the ASF dual-hosted git repository.
nehapawar pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git.
from 5da3433 Support for Update & Delete in ZooKeeper Browser and added
SQL Functions in SQL Editor autocomplete list (#5981)
add 41de9a6 Segment processing framework (#5934)
No new revisions were added by this update.
Summary of changes:
.../apache/pinot/core/minion/SegmentConverter.java | 2 +-
.../minion/rollup/MergeRollupSegmentConverter.java | 2 +-
.../core/minion/rollup/RollupRecordAggregator.java | 33 +-
.../core/minion/segment/MapperRecordReader.java| 1 +
.../processing/collector/Collector.java} | 31 +-
.../processing/collector/CollectorConfig.java | 109
.../processing/collector/CollectorFactory.java | 61 +++
.../processing/collector/ConcatCollector.java | 67 +++
.../processing/collector/GenericRowSorter.java | 90
.../processing/collector}/MaxValueAggregator.java | 17 +-
.../processing/collector/MinValueAggregator.java} | 29 +-
.../processing/collector/RollupCollector.java | 165 ++
.../processing/collector}/SumValueAggregator.java | 20 +-
.../processing/collector}/ValueAggregator.java | 12 +-
.../collector}/ValueAggregatorFactory.java | 22 +-
.../filter/FunctionEvaluatorRecordFilter.java | 46 ++
.../processing/filter/NoOpRecordFilter.java} | 17 +-
.../processing/filter/RecordFilter.java} | 13 +-
.../processing/filter/RecordFilterConfig.java | 89
.../processing/filter/RecordFilterFactory.java | 59 +++
.../processing/framework/SegmentConfig.java| 72 +++
.../processing/framework/SegmentMapper.java| 137 +
.../processing/framework/SegmentMapperConfig.java | 72 +++
.../framework/SegmentProcessorConfig.java | 181 +++
.../framework/SegmentProcessorFramework.java | 196
.../processing/framework/SegmentReducer.java | 133 +
.../processing/framework/SegmentReducerConfig.java | 61 +++
.../partitioner/ColumnValuePartitioner.java} | 23 +-
.../processing/partitioner/NoOpPartitioner.java} | 18 +-
.../processing/partitioner/Partitioner.java} | 13 +-
.../processing/partitioner/PartitionerConfig.java | 143 ++
.../processing/partitioner/PartitionerFactory.java | 88
.../partitioner/RoundRobinPartitioner.java}| 25 +-
.../partitioner/TableConfigPartitioner.java| 45 ++
.../partitioner/TransformFunctionPartitioner.java} | 25 +-
.../transformer/NoOpRecordTransformer.java}| 18 +-
.../processing/transformer}/RecordTransformer.java | 2 +-
.../transformer/RecordTransformerConfig.java | 71 +++
.../transformer/RecordTransformerFactory.java} | 25 +-
.../TransformFunctionRecordTransformer.java| 49 ++
.../processing/utils/SegmentProcessorUtils.java| 113 +
.../processing/framework/CollectorTest.java| 291 +++
.../processing/framework/GenericRowSorterTest.java | 77 +++
.../processing/framework/PartitionerTest.java | 194
.../processing/framework/RecordFilterTest.java | 83
.../framework/RecordTransformerTest.java | 86
.../processing/framework/SegmentMapperTest.java| 274 ++
.../framework/SegmentProcessingFrameworkTest.java | 551 +
.../processing/framework/SegmentReducerTest.java | 229 +
.../pinot/tools/admin/PinotAdministrator.java | 4 +-
.../command/SegmentProcessorFrameworkCommand.java | 116 +
.../processor/SegmentProcessorFrameworkSpec.java | 115 +
52 files changed, 4261 insertions(+), 154 deletions(-)
copy
pinot-core/src/main/java/org/apache/pinot/core/{minion/segment/RecordTransformer.java
=> segment/processing/collector/Collector.java} (57%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/collector/CollectorConfig.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/collector/CollectorFactory.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/collector/ConcatCollector.java
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/collector/GenericRowSorter.java
copy pinot-core/src/main/java/org/apache/pinot/core/{minion/rollup/aggregate
=> segment/processing/collector}/MaxValueAggregator.java (81%)
rename
pinot-core/src/main/java/org/apache/pinot/core/{minion/rollup/aggregate/MaxValueAggregator.java
=> segment/processing/collector/MinValueAggregator.java} (63%)
create mode 100644
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/collector/RollupCollector.java
rename pinot-core/src/main/java/org/apache/pinot/core/{minion/rollup/aggregate
=> segment/processing/collector}/SumValueAggregator.java (71%)
copy
[GitHub] [incubator-pinot] npawar merged pull request #5934: Segment processing framework
npawar merged pull request #5934: URL: https://github.com/apache/incubator-pinot/pull/5934 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Include branch name when publishing pinot release to bintray
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch check-branch-name
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit a3bc8b60c3d8c4e4844171d90dd9b3e65d25b4e4
Author: Jack Li(Analytics Engineering)
AuthorDate: Tue Sep 15 10:47:32 2020 -0700
Include branch name when publishing pinot release to bintray
---
.travis/.travis_nightly_build.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/.travis/.travis_nightly_build.sh b/.travis/.travis_nightly_build.sh
index 602a4bf..d33e6a2 100755
--- a/.travis/.travis_nightly_build.sh
+++ b/.travis/.travis_nightly_build.sh
@@ -20,6 +20,7 @@
if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
echo "Deploying to bintray"
+ echo "Current branch name: ${TRAVIS_BRANCH}"
BUILD_VERSION=$(grep -E "(.*)-SNAPSHOT" pom.xml | cut
-d'>' -f2 | cut -d'<' -f1 | cut -d'-' -f1)
echo "Current build version: $BUILD_VERSION${DEV_VERSION}"
@@ -27,5 +28,5 @@ if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
mvn versions:commit -q -B
# Deploy to bintray
- mvn deploy -s .travis/.ci.settings.xml -DskipTests -q
-DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-maven::default::'https://api.bintray.com/maven/linkedin/maven/pinot/;publish=1;override=1'
+ mvn deploy -s .travis/.ci.settings.xml -DscmBranch="${TRAVIS_BRANCH}"
-DskipTests -q -DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-maven::default::'https://api.bintray.com/maven/linkedin/maven/pinot/;publish=1;override=1'
fi
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch check-branch-name updated (c5905be -> a3bc8b6)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch check-branch-name in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard c5905be Validate branch name new a3bc8b6 Include branch name when publishing pinot release to bintray This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (c5905be) \ N -- N -- N refs/heads/check-branch-name (a3bc8b6) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis/.ci.settings.xml | 2 +- .travis/.travis_nightly_build.sh | 6 +++--- pom.xml | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 02/04: Fetch configs from cache rather than zookeeper
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a commit to branch controller_api_patch
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit f3472898ad5e81f0de3267c4b2333b629fc9d007
Author: Kartik Khare
AuthorDate: Tue Sep 15 19:56:11 2020 +0530
Fetch configs from cache rather than zookeeper
---
.../helix/core/PinotHelixResourceManager.java | 70 +-
1 file changed, 54 insertions(+), 16 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
index 1536128..917a777 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
@@ -25,7 +25,6 @@ import com.google.common.cache.CacheLoader;
import com.google.common.cache.LoadingCache;
import com.google.common.collect.BiMap;
import com.google.common.collect.HashBiMap;
-import com.google.common.collect.Lists;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
@@ -57,6 +56,8 @@ import org.apache.helix.InstanceType;
import org.apache.helix.PropertyKey;
import org.apache.helix.PropertyKey.Builder;
import org.apache.helix.ZNRecord;
+import org.apache.helix.api.listeners.ExternalViewChangeListener;
+import org.apache.helix.api.listeners.LiveInstanceChangeListener;
import org.apache.helix.model.CurrentState;
import org.apache.helix.model.ExternalView;
import org.apache.helix.model.HelixConfigScope;
@@ -103,6 +104,9 @@ import
org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssign
import
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignment;
import
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignmentFactory;
import
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignmentUtils;
+import
org.apache.pinot.controller.helix.core.listener.ClusterExternalViewChangeListener;
+import
org.apache.pinot.controller.helix.core.listener.ClusterInstanceConfigChangeListener;
+import
org.apache.pinot.controller.helix.core.listener.ClusterLiveInstanceChangeListener;
import
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager;
import org.apache.pinot.controller.helix.core.rebalance.RebalanceResult;
import org.apache.pinot.controller.helix.core.rebalance.TableRebalancer;
@@ -160,6 +164,9 @@ public class PinotHelixResourceManager {
private SegmentDeletionManager _segmentDeletionManager;
private PinotLLCRealtimeSegmentManager _pinotLLCRealtimeSegmentManager;
private TableCache _tableCache;
+ private ClusterInstanceConfigChangeListener
_clusterInstanceConfigChangeListener;
+ private ClusterLiveInstanceChangeListener _clusterLiveInstanceChangeListener;
+ private ClusterExternalViewChangeListener _clusterExternalViewChangeListener;
public PinotHelixResourceManager(String zkURL, String helixClusterName,
@Nullable String dataDir,
long externalViewOnlineToOfflineTimeoutMillis, boolean
isSingleTenantCluster, boolean enableBatchMessageMode,
@@ -223,6 +230,18 @@ public class PinotHelixResourceManager {
boolean caseInsensitive =
Boolean.parseBoolean(configs.get(Helix.ENABLE_CASE_INSENSITIVE_KEY)) || Boolean
.parseBoolean(configs.get(Helix.DEPRECATED_ENABLE_CASE_INSENSITIVE_KEY));
_tableCache = new TableCache(_propertyStore, caseInsensitive);
+
+_clusterInstanceConfigChangeListener = new
ClusterInstanceConfigChangeListener();
+_clusterLiveInstanceChangeListener = new
ClusterLiveInstanceChangeListener();
+_clusterExternalViewChangeListener = new
ClusterExternalViewChangeListener();
+try {
+ addConfigListeners(_clusterInstanceConfigChangeListener);
+ addLiveInstanceListeners(_clusterLiveInstanceChangeListener);
+ addExternalViewListeners(_clusterExternalViewChangeListener);
+} catch (Exception e) {
+ LOGGER.warn(
+ "Unable to add config listener in controller. This will result in
incorrect response from controller's broker API");
+}
}
/**
@@ -317,7 +336,7 @@ public class PinotHelixResourceManager {
* Returns the config for all the Helix instances in the cluster.
*/
public List getAllHelixInstanceConfigs() {
-return HelixHelper.getInstanceConfigs(_helixZkManager);
+return _clusterInstanceConfigChangeListener.getInstanceConfigs();
}
/**
@@ -359,14 +378,15 @@ public class PinotHelixResourceManager {
brokerTenantName = realtimeTableConfig.getTenantConfig().getBroker();
}
}
-return HelixHelper.getInstancesWithTag(_helixZkManager,
TagNameUtils.getBrokerTagForTenant(brokerTenantName));
+return
[incubator-pinot] branch controller_api_patch created (now 36babc3)
This is an automated email from the ASF dual-hosted git repository. kharekartik pushed a change to branch controller_api_patch in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 36babc3 change in constructors of listeners This branch includes the following new commits: new acd9397 Add listeners to cache zookeeper configs for helix new f347289 Fetch configs from cache rather than zookeeper new e30ffb9 Add change for Helix Configs new 36babc3 change in constructors of listeners The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/04: Add listeners to cache zookeeper configs for helix
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a commit to branch controller_api_patch
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit acd9397d28f1edd400c4b6fcdea8473a770d475e
Author: Kartik Khare
AuthorDate: Tue Sep 15 19:54:47 2020 +0530
Add listeners to cache zookeeper configs for helix
---
.../ClusterExternalViewChangeListener.java | 21 +++
.../ClusterInstanceConfigChangeListener.java | 24 ++
.../ClusterLiveInstanceChangeListener.java | 21 +++
3 files changed, 66 insertions(+)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterExternalViewChangeListener.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterExternalViewChangeListener.java
new file mode 100644
index 000..6400f47
--- /dev/null
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterExternalViewChangeListener.java
@@ -0,0 +1,21 @@
+package org.apache.pinot.controller.helix.core.listener;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.helix.NotificationContext;
+import org.apache.helix.api.listeners.ExternalViewChangeListener;
+import org.apache.helix.model.ExternalView;
+
+
+public class ClusterExternalViewChangeListener implements
ExternalViewChangeListener {
+ private List _externalViewList = new ArrayList<>();
+
+ @Override
+ public void onExternalViewChange(List externalViewList,
NotificationContext changeContext) {
+_externalViewList = externalViewList;
+ }
+
+ public List getExternalViewList() {
+return _externalViewList;
+ }
+}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterInstanceConfigChangeListener.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterInstanceConfigChangeListener.java
new file mode 100644
index 000..940230c
--- /dev/null
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterInstanceConfigChangeListener.java
@@ -0,0 +1,24 @@
+package org.apache.pinot.controller.helix.core.listener;
+
+import org.apache.helix.NotificationContext;
+import org.apache.helix.api.listeners.InstanceConfigChangeListener;
+import org.apache.helix.model.InstanceConfig;
+
+import java.util.ArrayList;
+import java.util.List;
+
+public class ClusterInstanceConfigChangeListener implements
InstanceConfigChangeListener {
+private List _instanceConfigs = new ArrayList<>();
+
+public ClusterInstanceConfigChangeListener() {
+}
+
+@Override
+public void onInstanceConfigChange(List instanceConfigs,
NotificationContext context) {
+_instanceConfigs = instanceConfigs;
+}
+
+public List getInstanceConfigs() {
+return _instanceConfigs;
+}
+}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java
new file mode 100644
index 000..777b41a
--- /dev/null
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java
@@ -0,0 +1,21 @@
+package org.apache.pinot.controller.helix.core.listener;
+
+import java.util.ArrayList;
+import org.apache.helix.NotificationContext;
+import org.apache.helix.api.listeners.LiveInstanceChangeListener;
+import org.apache.helix.model.LiveInstance;
+
+import java.util.List;
+
+public class ClusterLiveInstanceChangeListener implements
LiveInstanceChangeListener {
+private List _liveInstances = new ArrayList<>();
+
+@Override
+public void onLiveInstanceChange(List liveInstances,
NotificationContext changeContext) {
+_liveInstances = liveInstances;
+}
+
+public List getLiveInstances() {
+return _liveInstances;
+}
+}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 03/04: Add change for Helix Configs
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a commit to branch controller_api_patch
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit e30ffb9ecb2be33ad5e1d6161deb838884b08fe1
Author: Kartik Khare
AuthorDate: Wed Sep 16 01:08:33 2020 +0530
Add change for Helix Configs
---
.../ClusterExternalViewChangeListener.java | 21 -
.../ClusterInstanceConfigChangeListener.java | 10 +++-
.../ClusterLiveInstanceChangeListener.java | 27 --
3 files changed, 29 insertions(+), 29 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterExternalViewChangeListener.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterExternalViewChangeListener.java
deleted file mode 100644
index 6400f47..000
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterExternalViewChangeListener.java
+++ /dev/null
@@ -1,21 +0,0 @@
-package org.apache.pinot.controller.helix.core.listener;
-
-import java.util.ArrayList;
-import java.util.List;
-import org.apache.helix.NotificationContext;
-import org.apache.helix.api.listeners.ExternalViewChangeListener;
-import org.apache.helix.model.ExternalView;
-
-
-public class ClusterExternalViewChangeListener implements
ExternalViewChangeListener {
- private List _externalViewList = new ArrayList<>();
-
- @Override
- public void onExternalViewChange(List externalViewList,
NotificationContext changeContext) {
-_externalViewList = externalViewList;
- }
-
- public List getExternalViewList() {
-return _externalViewList;
- }
-}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterInstanceConfigChangeListener.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterInstanceConfigChangeListener.java
index 940230c..88bdd40 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterInstanceConfigChangeListener.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterInstanceConfigChangeListener.java
@@ -1,16 +1,21 @@
package org.apache.pinot.controller.helix.core.listener;
+import org.apache.helix.HelixManager;
import org.apache.helix.NotificationContext;
import org.apache.helix.api.listeners.InstanceConfigChangeListener;
import org.apache.helix.model.InstanceConfig;
import java.util.ArrayList;
import java.util.List;
+import org.apache.pinot.common.utils.helix.HelixHelper;
+
public class ClusterInstanceConfigChangeListener implements
InstanceConfigChangeListener {
+private HelixManager _helixManager;
private List _instanceConfigs = new ArrayList<>();
-public ClusterInstanceConfigChangeListener() {
+public ClusterInstanceConfigChangeListener(HelixManager helixManager) {
+_helixManager = helixManager;
}
@Override
@@ -19,6 +24,9 @@ public class ClusterInstanceConfigChangeListener implements
InstanceConfigChange
}
public List getInstanceConfigs() {
+if(_instanceConfigs.isEmpty()){
+_instanceConfigs = HelixHelper.getInstanceConfigs(_helixManager);
+}
return _instanceConfigs;
}
}
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java
index 777b41a..e3623c7 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java
@@ -1,21 +1,34 @@
package org.apache.pinot.controller.helix.core.listener;
import java.util.ArrayList;
+import org.apache.helix.HelixDataAccessor;
import org.apache.helix.NotificationContext;
import org.apache.helix.api.listeners.LiveInstanceChangeListener;
import org.apache.helix.model.LiveInstance;
import java.util.List;
+import org.apache.helix.PropertyKey.Builder;
+
public class ClusterLiveInstanceChangeListener implements
LiveInstanceChangeListener {
-private List _liveInstances = new ArrayList<>();
+ private HelixDataAccessor _helixDataAccessor;
+ private Builder _keyBuilder;
+ private List _liveInstances = new ArrayList<>();
-@Override
-public void onLiveInstanceChange(List liveInstances,
NotificationContext changeContext) {
-_liveInstances = liveInstances;
-}
+ public ClusterLiveInstanceChangeListener(HelixDataAccessor
helixDataAccessor, Builder keyBuilder) {
+_helixDataAccessor = helixDataAccessor;
+_keyBuilder = keyBuilder;
+ }
+
+ @Override
+ public void onLiveInstanceChange(List liveInstances,
[incubator-pinot] branch check-branch-name updated (6ba644e -> c5905be)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch check-branch-name in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 6ba644e Validate branch name new c5905be Validate branch name This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (6ba644e) \ N -- N -- N refs/heads/check-branch-name (c5905be) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis/.travis_nightly_build.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Validate branch name
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch check-branch-name
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit c5905be15f839add19c746cb23e2095d10fd0d1f
Author: Jack Li(Analytics Engineering)
AuthorDate: Tue Sep 15 10:47:32 2020 -0700
Validate branch name
---
.travis/.ci.settings.xml | 2 +-
.travis/.travis_nightly_build.sh | 7 ---
pom.xml | 2 +-
3 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/.travis/.ci.settings.xml b/.travis/.ci.settings.xml
index 587dd4f..debfe83 100644
--- a/.travis/.ci.settings.xml
+++ b/.travis/.ci.settings.xml
@@ -23,7 +23,7 @@
xmlns='http://maven.apache.org/SETTINGS/1.0.0'
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'>
- bintray-linkedin-maven
+ bintray-linkedin-test-repo
${env.MVN_DEPLOY_BINTRAY_USER}
${env.MVN_DEPLOY_BINTRAY_KEY}
diff --git a/.travis/.travis_nightly_build.sh b/.travis/.travis_nightly_build.sh
index 602a4bf..4e0b04a 100755
--- a/.travis/.travis_nightly_build.sh
+++ b/.travis/.travis_nightly_build.sh
@@ -18,8 +18,9 @@
# under the License.
#
-if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
+#if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
echo "Deploying to bintray"
+ echo "Current branch name: ${TRAVIS_BRANCH}"
BUILD_VERSION=$(grep -E "(.*)-SNAPSHOT" pom.xml | cut
-d'>' -f2 | cut -d'<' -f1 | cut -d'-' -f1)
echo "Current build version: $BUILD_VERSION${DEV_VERSION}"
@@ -27,5 +28,5 @@ if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
mvn versions:commit -q -B
# Deploy to bintray
- mvn deploy -s .travis/.ci.settings.xml -DskipTests -q
-DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-maven::default::'https://api.bintray.com/maven/linkedin/maven/pinot/;publish=1;override=1'
-fi
+ mvn deploy -s .travis/.ci.settings.xml -DscmBranch="${TRAVIS_BRANCH}"
-DskipTests -q -DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-test-repo::default::'https://api.bintray.com/maven/linkedin/test-repo/pinot/;publish=1;override=1'
+#fi
diff --git a/pom.xml b/pom.xml
index d69be3d..6888cf1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1368,7 +1368,7 @@
org.apache.maven.scm
maven-scm-provider-gitexe
- 1.9.4
+ 1.11.2
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Include branch name when publishing pinot release to bintray (#6019)
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new c42c740 Include branch name when publishing pinot release to bintray
(#6019)
c42c740 is described below
commit c42c74056ddea201328b0efb35302f460490b3a5
Author: Jialiang Li
AuthorDate: Tue Sep 15 13:16:53 2020 -0700
Include branch name when publishing pinot release to bintray (#6019)
Co-authored-by: Jack Li(Analytics Engineering)
---
.travis/.travis_nightly_build.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/.travis/.travis_nightly_build.sh b/.travis/.travis_nightly_build.sh
index 602a4bf..d33e6a2 100755
--- a/.travis/.travis_nightly_build.sh
+++ b/.travis/.travis_nightly_build.sh
@@ -20,6 +20,7 @@
if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
echo "Deploying to bintray"
+ echo "Current branch name: ${TRAVIS_BRANCH}"
BUILD_VERSION=$(grep -E "(.*)-SNAPSHOT" pom.xml | cut
-d'>' -f2 | cut -d'<' -f1 | cut -d'-' -f1)
echo "Current build version: $BUILD_VERSION${DEV_VERSION}"
@@ -27,5 +28,5 @@ if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
mvn versions:commit -q -B
# Deploy to bintray
- mvn deploy -s .travis/.ci.settings.xml -DskipTests -q
-DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-maven::default::'https://api.bintray.com/maven/linkedin/maven/pinot/;publish=1;override=1'
+ mvn deploy -s .travis/.ci.settings.xml -DscmBranch="${TRAVIS_BRANCH}"
-DskipTests -q -DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-maven::default::'https://api.bintray.com/maven/linkedin/maven/pinot/;publish=1;override=1'
fi
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli opened a new pull request #6019: Include branch name when publishing pinot release to bintray
jackjlli opened a new pull request #6019: URL: https://github.com/apache/incubator-pinot/pull/6019 ## Description This PR includes branch name in the manifest when publishing pinot release to bintray. As someone mentioned in https://github.com/mojohaus/buildnumber-maven-plugin/issues/53: > If you are using Jenkins or some automated CI tool, Its most likely that the commit was checked out in a detached state. In detached state, HEAD won't have the branch information That's why the branch name in previous release is shown `UNKNOWN`. Sample info after this change from https://travis-ci.org/github/apache/incubator-pinot/builds/727461249: ``` Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: travis Build-Jdk: 1.8.0_151 Specification-Title: Pinot Controller Specification-Version: 0.5.0-dev-18752 Specification-Vendor: Apache Software Foundation Build-Time: 2020-09-15T18:31:31Z Implementation-Branch: check-branch-name Implementation-Title: pinot-controller Implementation-Vendor: Apache Software Foundation Implementation-Vendor-Id: org.apache.pinot Implementation-Version: 0.5.0-dev-18752-c5905be15f839add19c746cb23e209 5d10fd0d1f ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps opened a new pull request #6020: Add Caching in Controller Broker API
KKcorps opened a new pull request #6020: URL: https://github.com/apache/incubator-pinot/pull/6020 Currently, Controller does multiple calls to zookeeper to fetch the list of brokers per tenant and table. This PR aims to cache the instance configs of zookeeper so that the calls don't go always. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli merged pull request #6019: Include branch name when publishing pinot release to bintray
jackjlli merged pull request #6019: URL: https://github.com/apache/incubator-pinot/pull/6019 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Validate branch name
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch check-branch-name
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 6ba644e391e795b420f043873fd9e39ec9f1e47d
Author: Jack Li(Analytics Engineering)
AuthorDate: Tue Sep 15 10:47:32 2020 -0700
Validate branch name
---
.travis/.ci.settings.xml | 2 +-
.travis/.travis_nightly_build.sh | 6 +++---
pom.xml | 2 +-
3 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/.travis/.ci.settings.xml b/.travis/.ci.settings.xml
index 587dd4f..debfe83 100644
--- a/.travis/.ci.settings.xml
+++ b/.travis/.ci.settings.xml
@@ -23,7 +23,7 @@
xmlns='http://maven.apache.org/SETTINGS/1.0.0'
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'>
- bintray-linkedin-maven
+ bintray-linkedin-test-repo
${env.MVN_DEPLOY_BINTRAY_USER}
${env.MVN_DEPLOY_BINTRAY_KEY}
diff --git a/.travis/.travis_nightly_build.sh b/.travis/.travis_nightly_build.sh
index 602a4bf..5595c32 100755
--- a/.travis/.travis_nightly_build.sh
+++ b/.travis/.travis_nightly_build.sh
@@ -18,7 +18,7 @@
# under the License.
#
-if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
+#if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
echo "Deploying to bintray"
BUILD_VERSION=$(grep -E "(.*)-SNAPSHOT" pom.xml | cut
-d'>' -f2 | cut -d'<' -f1 | cut -d'-' -f1)
@@ -27,5 +27,5 @@ if [ -n "${DEPLOY_BUILD_OPTS}" ]; then
mvn versions:commit -q -B
# Deploy to bintray
- mvn deploy -s .travis/.ci.settings.xml -DskipTests -q
-DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-maven::default::'https://api.bintray.com/maven/linkedin/maven/pinot/;publish=1;override=1'
-fi
+ mvn deploy -s .travis/.ci.settings.xml -DskipTests -q
-DretryFailedDeploymentCount=5
-DaltDeploymentRepository=bintray-linkedin-test-repo::default::'https://api.bintray.com/maven/linkedin/test-repo/pinot/;publish=1;override=1'
+#fi
diff --git a/pom.xml b/pom.xml
index d69be3d..6888cf1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1368,7 +1368,7 @@
org.apache.maven.scm
maven-scm-provider-gitexe
- 1.9.4
+ 1.11.2
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch check-branch-name created (now 6ba644e)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch check-branch-name in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 6ba644e Validate branch name This branch includes the following new commits: new 6ba644e Validate branch name The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 04/04: change in constructors of listeners
This is an automated email from the ASF dual-hosted git repository.
kharekartik pushed a commit to branch controller_api_patch
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 36babc3ef54be37809e8b0e14119080a3aada5ca
Author: Kartik Khare
AuthorDate: Wed Sep 16 01:10:38 2020 +0530
change in constructors of listeners
---
.../controller/helix/core/PinotHelixResourceManager.java| 13 ++---
1 file changed, 2 insertions(+), 11 deletions(-)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
index 917a777..27920db 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java
@@ -104,7 +104,6 @@ import
org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssign
import
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignment;
import
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignmentFactory;
import
org.apache.pinot.controller.helix.core.assignment.segment.SegmentAssignmentUtils;
-import
org.apache.pinot.controller.helix.core.listener.ClusterExternalViewChangeListener;
import
org.apache.pinot.controller.helix.core.listener.ClusterInstanceConfigChangeListener;
import
org.apache.pinot.controller.helix.core.listener.ClusterLiveInstanceChangeListener;
import
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager;
@@ -166,7 +165,6 @@ public class PinotHelixResourceManager {
private TableCache _tableCache;
private ClusterInstanceConfigChangeListener
_clusterInstanceConfigChangeListener;
private ClusterLiveInstanceChangeListener _clusterLiveInstanceChangeListener;
- private ClusterExternalViewChangeListener _clusterExternalViewChangeListener;
public PinotHelixResourceManager(String zkURL, String helixClusterName,
@Nullable String dataDir,
long externalViewOnlineToOfflineTimeoutMillis, boolean
isSingleTenantCluster, boolean enableBatchMessageMode,
@@ -231,13 +229,11 @@ public class PinotHelixResourceManager {
.parseBoolean(configs.get(Helix.DEPRECATED_ENABLE_CASE_INSENSITIVE_KEY));
_tableCache = new TableCache(_propertyStore, caseInsensitive);
-_clusterInstanceConfigChangeListener = new
ClusterInstanceConfigChangeListener();
-_clusterLiveInstanceChangeListener = new
ClusterLiveInstanceChangeListener();
-_clusterExternalViewChangeListener = new
ClusterExternalViewChangeListener();
+_clusterInstanceConfigChangeListener = new
ClusterInstanceConfigChangeListener(_helixZkManager);
+_clusterLiveInstanceChangeListener = new
ClusterLiveInstanceChangeListener(_helixDataAccessor, _keyBuilder);
try {
addConfigListeners(_clusterInstanceConfigChangeListener);
addLiveInstanceListeners(_clusterLiveInstanceChangeListener);
- addExternalViewListeners(_clusterExternalViewChangeListener);
} catch (Exception e) {
LOGGER.warn(
"Unable to add config listener in controller. This will result in
incorrect response from controller's broker API");
@@ -2215,11 +2211,6 @@ public class PinotHelixResourceManager {
_helixZkManager.addLiveInstanceChangeListener(liveInstanceChangeListener);
}
- public void addExternalViewListeners(ExternalViewChangeListener
externalViewChangeListener)
- throws Exception {
-_helixZkManager.addExternalViewChangeListener(externalViewChangeListener);
- }
-
/**
* Provides admin endpoints for the provided data instances
* @param instances instances for which to read endpoints
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Add a length limit of 512 to the properties stored in the segment metadata (#6008)
This is an automated email from the ASF dual-hosted git repository.
jackie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new b2df8ea Add a length limit of 512 to the properties stored in the
segment metadata (#6008)
b2df8ea is described below
commit b2df8ea51180df272cc7e67b91ac2f7827db4b1a
Author: Xiaotian (Jackie) Jiang <1751+jackie-ji...@users.noreply.github.com>
AuthorDate: Tue Sep 15 15:04:04 2020 -0700
Add a length limit of 512 to the properties stored in the segment metadata
(#6008)
---
.../core/segment/creator/impl/SegmentColumnarIndexCreator.java | 7 +++
1 file changed, 7 insertions(+)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
index 4489dc8..e3fda09 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/SegmentColumnarIndexCreator.java
@@ -78,6 +78,9 @@ import static
org.apache.pinot.core.segment.creator.impl.V1Constants.MetadataKey
public class SegmentColumnarIndexCreator implements SegmentCreator {
// TODO Refactor class name to match interface name
private static final Logger LOGGER =
LoggerFactory.getLogger(SegmentColumnarIndexCreator.class);
+ // Allow at most 512 characters for the metadata property
+ private static final int METADATA_PROPERTY_LENGTH_LIMIT = 512;
+
private SegmentGeneratorConfig config;
private Map indexCreationInfoMap;
private Map _dictionaryCreatorMap = new
HashMap<>();
@@ -562,6 +565,7 @@ public class SegmentColumnarIndexCreator implements
SegmentCreator {
* Helper method to check whether the given value is a valid property value.
* Value is invalid iff:
*
+ * It contains more than 512 characters
* It contains leading/trailing whitespace
* It contains list separator (',')
*
@@ -572,6 +576,9 @@ public class SegmentColumnarIndexCreator implements
SegmentCreator {
if (length == 0) {
return true;
}
+if (length > METADATA_PROPERTY_LENGTH_LIMIT) {
+ return false;
+}
if (Character.isWhitespace(value.charAt(0)) ||
Character.isWhitespace(value.charAt(length - 1))) {
return false;
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6008: Add a length limit of 512 to the properties stored in the segment metadata
Jackie-Jiang merged pull request #6008: URL: https://github.com/apache/incubator-pinot/pull/6008 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6021: List of partitioners in SegmentProcessorFramework
Jackie-Jiang commented on a change in pull request #6021:
URL: https://github.com/apache/incubator-pinot/pull/6021#discussion_r489090953
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/framework/SegmentMapper.java
##
@@ -100,8 +110,11 @@ public void map()
}
// Partitioning
- // TODO: 2 step partitioner. 1) Apply custom partitioner 2) Apply table
config partitioner. Combine both to get final partition.
- String partition = _partitioner.getPartition(reusableRow);
+ int p = 0;
+ for (Partitioner partitioner : _partitioners) {
+partitions[p++] = partitioner.getPartition(reusableRow);
+ }
+ String partition = StringUtil.join("_", partitions);
Review comment:
I don't think the framework relies on this to extract the partition
info. But agree on introducing a constant for it
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5960: LLC stopped realtime ingestion due to deadlock caused by out of order Helix state transition events (Pinot 0.4+)
mcvsubbu commented on issue #5960: URL: https://github.com/apache/incubator-pinot/issues/5960#issuecomment-693104377 It is useful add some notes here on this. The reason the semaphore was put in place was because Kafka consumers had an issue with dynamically creating new kafka metrics when a new consumer was created on the same partition while keeping the old consumer open. This led to metrics proliferation. From kafka (or any other stream) point of view, if they are trying to emit per-partition consumer metrics, it probably makes sense that they add different metrics for each consumer so as to get the health of the system correctly. So, we need to handle it somehow. One way could be that when we implement an internal "queue" of state transitions, holding the consuming transition until the ONLINE transition is received. Of course, we will have to respond back OK for the consuming state transition, but just not allow our software to handle it. A little tricky to implement, and sounds somewhat hacky to me. Also has a downside that we will be responding OK without actually handling the state transition. Helix does not provide a way to mark a transition in ERROR from the participant. We could ask for that feature (useful for other things as well), and then we can mark the consumer in ERROR state if it later fails to transition correctly. A simple work-around in the short/medium term may be we stop realtime table creation in the controller if the max-threads is not set to at least 2 (we don't expect to have more than one outstanding state transition for the same partition). I am open to hearing other ideas on how to fix the deadlock. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #6021: List of partitioners in SegmentProcessorFramework
npawar commented on a change in pull request #6021:
URL: https://github.com/apache/incubator-pinot/pull/6021#discussion_r489102318
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/framework/SegmentMapper.java
##
@@ -100,8 +110,11 @@ public void map()
}
// Partitioning
- // TODO: 2 step partitioner. 1) Apply custom partitioner 2) Apply table
config partitioner. Combine both to get final partition.
- String partition = _partitioner.getPartition(reusableRow);
+ int p = 0;
+ for (Partitioner partitioner : _partitioners) {
+partitions[p++] = partitioner.getPartition(reusableRow);
+ }
+ String partition = StringUtil.join("_", partitions);
Review comment:
Actually, it is not significant at all. It can be changed, and is not
used by any other components. It won't even matter beyond the scope of that
joiner line. And hence I don't think it needs to be scoped out of this class,
or even out of this method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar opened a new pull request #6021: List of partitioners in SegmentProcessorFramework
npawar opened a new pull request #6021: URL: https://github.com/apache/incubator-pinot/pull/6021 Changing SegmentPrpcessorFramework config to take List of Partitioners. This is to account for table config's partitioning, which might need to be applied regardless of other partitioning configured. For example, we want to partition by date so as to align segments by time, but we also want to apply Murmur on some id column each to further partition and record in segment metadata. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #6017: Table indexing config validation
npawar commented on a change in pull request #6017:
URL: https://github.com/apache/incubator-pinot/pull/6017#discussion_r489027314
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/util/TableConfigUtils.java
##
@@ -225,4 +229,54 @@ private static void validateTierConfigList(@Nullable
List tierConfig
}
}
}
+
+ private static void validateIndexingConfig(@Nullable IndexingConfig
indexingConfig, @Nullable Schema schema) {
+if (indexingConfig == null || schema == null) {
+ return;
+}
+Map columnNameToConfigMap = new HashMap<>();
+
+if (indexingConfig.getBloomFilterColumns() != null) {
+ for (String columnName : indexingConfig.getBloomFilterColumns()) {
+columnNameToConfigMap.put(columnName, "Bloom Filter Config");
+ }
+}
+if (indexingConfig.getInvertedIndexColumns() != null) {
+ for (String columnName : indexingConfig.getInvertedIndexColumns()) {
+columnNameToConfigMap.put(columnName, "Inverted Index Config");
+ }
+}
+if (indexingConfig.getNoDictionaryColumns() != null) {
+ for (String columnName : indexingConfig.getNoDictionaryColumns()) {
+columnNameToConfigMap.put(columnName, "No Dictionary Column Config");
+ }
+}
+if (indexingConfig.getOnHeapDictionaryColumns() != null) {
+ for (String columnName : indexingConfig.getOnHeapDictionaryColumns()) {
+columnNameToConfigMap.put(columnName, "On Heap Dictionary Column
Config");
+ }
+}
+if (indexingConfig.getRangeIndexColumns() != null) {
+ for (String columnName : indexingConfig.getRangeIndexColumns()) {
+columnNameToConfigMap.put(columnName, "Range Column Config");
+ }
+}
+if (indexingConfig.getSortedColumn() != null) {
+ for (String columnName : indexingConfig.getSortedColumn()) {
+columnNameToConfigMap.put(columnName, "Sorted Column Config");
+ }
+}
+if (indexingConfig.getVarLengthDictionaryColumns() != null) {
+ for (String columnName : indexingConfig.getVarLengthDictionaryColumns())
{
+columnNameToConfigMap.put(columnName, "Var Length Column Config");
+ }
+}
+
Review comment:
can we include star tree index columns here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] add labeler into yaml (#6007)
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new a123c60 [TE] add labeler into yaml (#6007)
a123c60 is described below
commit a123c60f63e3fc78fb691b380097aaf1f8c4d4b8
Author: Vincent Chen
AuthorDate: Tue Sep 15 15:40:34 2020 -0700
[TE] add labeler into yaml (#6007)
This PR is second PR for severity-based alert feature, including the logic
of parsing labeler configuration and constructing the detection pipelines based
on the YAML.
---
.../components/ThresholdSeverityLabeler.java | 16 ++---
.../thirdeye/detection/spi/components/Labeler.java | 9 ++-
.../detection/wrapper/AnomalyLabelerWrapper.java | 13 +++-
.../wrapper/ChildKeepingMergeWrapper.java | 6 +-
.../yaml/translator/DetectionConfigTranslator.java | 54 --
.../builder/DataQualityPropertiesBuilder.java | 2 +-
.../builder/DetectionConfigPropertiesBuilder.java | 34 +
.../builder/DetectionPropertiesBuilder.java| 23 --
.../detection/detection-config-schema.json | 3 +
.../thirdeye/detection/DetectionTestUtils.java | 16 +++--
.../components/ThresholdSeverityLabelerTest.java | 2 +-
.../wrapper/ChildKeepingMergeWrapperTest.java | 24 +++
.../translator/DetectionConfigTranslatorTest.java | 3 +
.../compositePipelineTranslatorTestResult-1.json | 84 +++---
.../yaml/translator/pipeline-config-1.yaml | 19 +
15 files changed, 223 insertions(+), 85 deletions(-)
diff --git
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/ThresholdSeverityLabeler.java
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/ThresholdSeverityLabeler.java
index ff04015..5f31ccf 100644
---
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/ThresholdSeverityLabeler.java
+++
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/components/ThresholdSeverityLabeler.java
@@ -19,6 +19,7 @@
package org.apache.pinot.thirdeye.detection.components;
+import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
@@ -46,7 +47,8 @@ public class ThresholdSeverityLabeler implements
Labeler severityMap;
@Override
- public void label(List anomalies) {
+ public Map
label(List anomalies) {
+Map res = new HashMap<>();
for (MergedAnomalyResultDTO anomaly : anomalies) {
double currVal = anomaly.getAvgCurrentVal();
double baseVal = anomaly.getAvgBaselineVal();
@@ -59,18 +61,12 @@ public class ThresholdSeverityLabeler implements
Labeler entry :
severityMap.entrySet()) {
if (deviation >= entry.getValue().change || duration >=
entry.getValue().duration) {
- if (anomaly.getSeverityLabel() != entry.getKey()) {
-// find the severity from highest to lowest
-if (anomaly.getId() != null &&
anomaly.getSeverityLabel().compareTo(entry.getKey()) > 0) {
- // only set renotify if the anomaly exists and its severity gets
higher
- anomaly.setRenotify(true);
-}
-anomaly.setSeverityLabel(entry.getKey());
-break;
- }
+ res.put(anomaly, entry.getKey());
+ break;
}
}
}
+return res;
}
@Override
diff --git
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/spi/components/Labeler.java
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/spi/components/Labeler.java
index 61541a4..77e4316 100644
---
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/spi/components/Labeler.java
+++
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/spi/components/Labeler.java
@@ -20,14 +20,17 @@
package org.apache.pinot.thirdeye.detection.spi.components;
import java.util.List;
+import java.util.Map;
+import org.apache.pinot.thirdeye.anomaly.AnomalySeverity;
import org.apache.pinot.thirdeye.datalayer.dto.MergedAnomalyResultDTO;
import org.apache.pinot.thirdeye.detection.spec.AbstractSpec;
public interface Labeler extends BaseComponent {
/**
- * add or modify labels of anomalies in place
- * @param anomalies
+ * Calculate the severity for list of anomalies
+ * @param anomalies input anoamlies
+ * @return mapping from anomaly to severity
*/
- void label(List anomalies);
+ Map
label(List anomalies);
}
diff --git
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/wrapper/AnomalyLabelerWrapper.java
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/wrapper/AnomalyLabelerWrapper.java
index 5d59e9b..9237fb6 100644
---
[GitHub] [incubator-pinot] jihaozh merged pull request #6007: [TE] add labeler into yaml
jihaozh merged pull request #6007: URL: https://github.com/apache/incubator-pinot/pull/6007 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] chenboat commented on issue #5960: LLC stopped realtime ingestion due to deadlock caused by out of order Helix state transition events (Pinot 0.4+)
chenboat commented on issue #5960: URL: https://github.com/apache/incubator-pinot/issues/5960#issuecomment-693096957 Thanks. The issue was resolved by now. But it will be great to explicitly exposed this config to Pinot installation. It is also worthwhile to fix the deadlock when maxThreads == 1. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #6022: Add IN_SUBQUERY support
Jackie-Jiang opened a new pull request #6022: URL: https://github.com/apache/incubator-pinot/pull/6022 ## Description Add `IN_SUBQUERY` transform function to support `IDSET` aggregation function as the subquery. The subquery is handled as a separate query on broker side. E.g. The following 2 queries can be combined into one query: `SELECT ID_SET(col) FROM table WHERE date = 20200901` `SELECT DISTINCT_COUNT(col), date FROM table WHERE IN_ID_SET(col, '') = 1 GROUP BY date` -> `SELECT DISTINCT_COUNT(col), date FROM table WHERE IN_SUBQUERY(col, 'SELECT ID_SET(col) FROM table WHERE date = 20200901') = 1 GROUP BY date` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6021: List of partitioners in SegmentProcessorFramework
mcvsubbu commented on a change in pull request #6021:
URL: https://github.com/apache/incubator-pinot/pull/6021#discussion_r489089813
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/framework/SegmentMapper.java
##
@@ -100,8 +110,11 @@ public void map()
}
// Partitioning
- // TODO: 2 step partitioner. 1) Apply custom partitioner 2) Apply table
config partitioner. Combine both to get final partition.
- String partition = _partitioner.getPartition(reusableRow);
+ int p = 0;
+ for (Partitioner partitioner : _partitioners) {
+partitions[p++] = partitioner.getPartition(reusableRow);
+ }
+ String partition = StringUtil.join("_", partitions);
Review comment:
The `"_"` here is very significant, right? It cannot be changed, and has
to be used the same way across multiple components. Could you please declare it
as. a final string in some Constants class as a partition separator or
something?
And then re-use in tests
thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] lgo opened a new pull request #6023: add `@types/react-router` dep
lgo opened a new pull request #6023: URL: https://github.com/apache/incubator-pinot/pull/6023 ## Description Before, `@types/react-router` was unlisted. Because `@types/react-router-dom` only has `@types/react-router: "*"` defined, it is possible the wrong version is pulled. When this happens and a different major version is used, `npm run-script build` would fail during maven builds over type errors. After this, the correct version is now explicitly indicated to ensure the right type version. For good measure, I hoisted the other `@types` dependencies to the non-dev dependencies. I'm not entirely sure what specific situations this happens in, but for my build environment I was getting a different major version and maven builds were failing. I suspect this is likely because of using an internal registry. ## Upgrade Notes N/A ## Release Notes N/A ## Documentation N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6021: List of partitioners in SegmentProcessorFramework
mcvsubbu commented on a change in pull request #6021:
URL: https://github.com/apache/incubator-pinot/pull/6021#discussion_r489138013
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/framework/SegmentMapper.java
##
@@ -100,8 +110,11 @@ public void map()
}
// Partitioning
- // TODO: 2 step partitioner. 1) Apply custom partitioner 2) Apply table
config partitioner. Combine both to get final partition.
- String partition = _partitioner.getPartition(reusableRow);
+ int p = 0;
+ for (Partitioner partitioner : _partitioners) {
+partitions[p++] = partitioner.getPartition(reusableRow);
+ }
+ String partition = StringUtil.join("_", partitions);
Review comment:
We are partitioning the data, and the brokers have to construct the same
partition id in the same order of columns and with the same partition function
right?
Also, what is the use case for partitioning on more than one column?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (a123c60 -> 07666f9)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from a123c60 [TE] add labeler into yaml (#6007) add 07666f9 bump paranamer (2.6 -> 2.8) (#6024) No new revisions were added by this update. Summary of changes: LICENSE-binary | 2 +- pom.xml| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #6024: bump paranamer (2.6 -> 2.8) to fix Spark failures (on 2.4.4 scala 2.12)
fx19880617 merged pull request #6024: URL: https://github.com/apache/incubator-pinot/pull/6024 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] lgo opened a new pull request #6024: bump paranamer (2.6 -> 2.8)
lgo opened a new pull request #6024: URL: https://github.com/apache/incubator-pinot/pull/6024 ## Description Before on paranamer 2.6, there was an incompatibility with certain JDK 8 features as mentioned in https://github.com/paul-hammant/paranamer/issues/17. This seemed to happen in particular because of the use of a particular Spark version (2.4.4 on scala 2.12). When running the Spark ingestion job, it was failing at some point during execution. Also, is there any special procedure to update `LICENSE-binary`, other than edit the version number? Specifically, the error stack trace was the following. ``` 21:50:27.454 [Driver] ERROR org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand - Got exception to kick off standalone data ingestion job - -- | java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.spark.SparkSegmentGenerationJobRunner | at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:137) ~[eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:110) ~[eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:123) [eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main(LaunchDataIngestionJobCommand.java:65) [eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_265] | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_265] | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_265] | at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_265] | at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684) [eojo_0.org_apache_spark_spark_shaded_distro_2_12.jar:?] | Caused by: java.lang.ArrayIndexOutOfBoundsException: 28499 | at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.accept(BytecodeReadingParanamer.java:563) ~[eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at com.thoughtworks.paranamer.BytecodeReadingParanamer$ClassReader.access$200(BytecodeReadingParanamer.java:338) ~[eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at com.thoughtworks.paranamer.BytecodeReadingParanamer.lookupParameterNames(BytecodeReadingParanamer.java:103) ~[eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at com.thoughtworks.paranamer.CachingParanamer.lookupParameterNames(CachingParanamer.java:79) ~[eojo_0.org_apache_pinot_pinot_distribution_jar_shaded.jar:0.5.0-2020-08-13-SNAPSHOT-cc55ad2fc0e3f6d0da769af286ddcb33d92ef229] | at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.getCtorParams(BeanIntrospector.scala:44) ~[ecjc_0.com_fasterxml_jackson_module_jackson_module_scala_2_12.jar:?] | at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1(BeanIntrospector.scala:58) ~[ecjc_0.com_fasterxml_jackson_module_jackson_module_scala_2_12.jar:?] | at com.fasterxml.jackson.module.scala.introspect.BeanIntrospector$.$anonfun$apply$1$adapted(BeanIntrospector.scala:58) ~[ecjc_0.com_fasterxml_jackson_module_jackson_module_scala_2_12.jar:?] | at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245) ~[eis_0.scala-library-2.12.10.jar:?] | at scala.collection.Iterator.foreach(Iterator.scala:941) ~[eis_0.scala-library-2.12.10.jar:?] | at scala.collection.Iterator.foreach$(Iterator.scala:941) ~[eis_0.scala-library-2.12.10.jar:?] | at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) ~[eis_0.scala-library-2.12.10.jar:?] | at scala.collection.IterableLike.foreach(IterableLike.scala:74) ~[eis_0.scala-library-2.12.10.jar:?] | at scala.collection.IterableLike.foreach$(IterableLike.scala:73) ~[eis_0.scala-library-2.12.10.jar:?] | at scala.collection.AbstractIterable.foreach(Iterable.scala:56) ~[eis_0.scala-library-2.12.10.jar:?] | at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
