[GitHub] [incubator-pinot] lgo opened a new issue #6018: APIs (like /instances) are inconsistently structured
lgo opened a new issue #6018:
URL: https://github.com/apache/incubator-pinot/issues/6018
Many APIs do not have consistent or easy to use resource structures. For
example, `GET /instances/{instanceName}` may return data like
```
{
"instanceName": "Broker_mybrokerhostname_8099",
"hostName": "Broker_mybrokerhostname",
"enabled": false,
"port": "8099",
"tags": [
"DefaultTenant_BROKER"
],
"pools": null
}
```
Meanwhile `PUT /instances{instanceName}` requires `host` (not `hostname`)
and a `type` field (`BROKER`, `SERVER`, `CONTROLLER`).
Mismatched resources like this make the APIs complex to work with and even
harder to use by hand.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6012: Configurable segment generation job parallelism for Hadoop and Spark
Jackie-Jiang commented on a change in pull request #6012:
URL: https://github.com/apache/incubator-pinot/pull/6012#discussion_r488393909
##
File path:
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
##
@@ -230,7 +230,10 @@ public void run()
if (hadoopTokenFileLocation != null) {
jobConf.set("mapreduce.job.credentials.binary",
hadoopTokenFileLocation);
}
- jobConf.setInt(JobContext.NUM_MAPS, numDataFiles);
+
+ int jobParallelism =
+ (_spec.getSegmentCreationJobParallelism() > 0) ?
_spec.getSegmentCreationJobParallelism() : numDataFiles;
Review comment:
Should we use min of configured parallelism and the `numDataFiles`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6012: Configurable segment generation job parallelism for Hadoop and Spark
Jackie-Jiang commented on a change in pull request #6012:
URL: https://github.com/apache/incubator-pinot/pull/6012#discussion_r488393909
##
File path:
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/src/main/java/org/apache/pinot/plugin/ingestion/batch/hadoop/HadoopSegmentGenerationJobRunner.java
##
@@ -230,7 +230,10 @@ public void run()
if (hadoopTokenFileLocation != null) {
jobConf.set("mapreduce.job.credentials.binary",
hadoopTokenFileLocation);
}
- jobConf.setInt(JobContext.NUM_MAPS, numDataFiles);
+
+ int jobParallelism =
+ (_spec.getSegmentCreationJobParallelism() > 0) ?
_spec.getSegmentCreationJobParallelism() : numDataFiles;
Review comment:
Should we use max of configured parallelism and the `numDataFiles`?
##
File path:
pinot-plugins/pinot-batch-ingestion/pinot-batch-ingestion-standalone/src/main/resources/segmentCreationAndTarPushJobSpec.yaml
##
@@ -28,6 +28,7 @@ includeFileNamePattern: 'glob:**/*.parquet'
excludeFileNamePattern: 'glob:**/*.avro'
outputDirURI: 'file:///path/to/output'
overwriteOutput: true
+parallelism: 1
Review comment:
1?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] icefury71 opened a new pull request #6017: Table indexing config validation
icefury71 opened a new pull request #6017: URL: https://github.com/apache/incubator-pinot/pull/6017 Related to #5942 ## Description This is to ensure any column name mentioned in the table indexing config is valid (i.e. it exists in the corresponding schema). Also checking for space in the table name. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) No Does this PR fix a zero-downtime upgrade introduced earlier? No Does this PR otherwise need attention when creating release notes? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] harleyjj opened a new pull request #6016: [TE] frontend - harleyjj/error-msgs - focus on errors in preview and …
harleyjj opened a new pull request #6016: URL: https://github.com/apache/incubator-pinot/pull/6016 …detection validation This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5934: Segment processing framework
Jackie-Jiang commented on a change in pull request #5934:
URL: https://github.com/apache/incubator-pinot/pull/5934#discussion_r488337421
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/utils/SegmentProcessorUtils.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.processing.utils;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.generic.GenericData;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * Helper util methods for SegmentProcessorFramework
+ */
+public final class SegmentProcessorUtils {
+
+ private SegmentProcessorUtils() {
+ }
+
+ /**
+ * Convert a GenericRow to an avro GenericRecord
+ */
+ public static GenericData.Record convertGenericRowToAvroRecord(GenericRow
genericRow,
+ GenericData.Record reusableRecord) {
+for (String field : genericRow.getFieldToValueMap().keySet()) {
Review comment:
The value in `fieldToValueMap` should be the default value if the
original value is `null`. IMO we should put `null` instead of default value in
the `GenericData.Record` so that the `null` values can be populated to the new
segment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #5934: Segment processing framework
npawar commented on a change in pull request #5934:
URL: https://github.com/apache/incubator-pinot/pull/5934#discussion_r488326079
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/partitioner/PartitionerFactory.java
##
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.processing.partitioner;
+
+import com.google.common.base.Preconditions;
+
+
+/**
+ * Factory for Partitioner and PartitionFilter
+ */
+public final class PartitionerFactory {
+
+ private PartitionerFactory() {
+
+ }
+
+ public enum PartitionerType {
+NO_OP, ROW_HASH, COLUMN_VALUE, TRANSFORM_FUNCTION, TABLE_PARTITION_CONFIG
+ }
+
+ /**
+ * Construct a Partitioner using the PartitioningConfig
+ */
+ public static Partitioner getPartitioner(PartitioningConfig config) {
+
+Partitioner partitioner = null;
+switch (config.getPartitionerType()) {
+ case NO_OP:
+partitioner = new NoOpPartitioner();
+break;
+ case ROW_HASH:
+Preconditions
+.checkState(config.getNumPartitions() > 0, "Must provide
numPartitions > 0 for ROW_HASH partitioner");
+partitioner = new RowHashPartitioner(config.getNumPartitions());
+break;
+ case COLUMN_VALUE:
+Preconditions.checkState(config.getColumnName() != null, "Must provide
columnName for COLUMN_VALUE partitioner");
+partitioner = new ColumnValuePartitioner(config.getColumnName());
+break;
+ case TRANSFORM_FUNCTION:
+Preconditions.checkState(config.getTransformFunction() != null,
+"Must provide transformFunction for TRANSFORM_FUNCTION
partitioner");
+partitioner = new
TransformFunctionPartitioner(config.getTransformFunction());
+break;
+ case TABLE_PARTITION_CONFIG:
Review comment:
Yes. Seunghyun pointed out similar thing. Discussion below.
I'll be changing the partitioner to a 2 step partitioner in the next PR
(have put TODO in code and descritpion)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on a change in pull request #5934: Segment processing framework
npawar commented on a change in pull request #5934:
URL: https://github.com/apache/incubator-pinot/pull/5934#discussion_r488325887
##
File path:
pinot-core/src/main/java/org/apache/pinot/core/segment/processing/utils/SegmentProcessorUtils.java
##
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.segment.processing.utils;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.avro.Schema;
+import org.apache.avro.SchemaBuilder;
+import org.apache.avro.generic.GenericData;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.readers.GenericRow;
+
+
+/**
+ * Helper util methods for SegmentProcessorFramework
+ */
+public final class SegmentProcessorUtils {
+
+ private SegmentProcessorUtils() {
+ }
+
+ /**
+ * Convert a GenericRow to an avro GenericRecord
+ */
+ public static GenericData.Record convertGenericRowToAvroRecord(GenericRow
genericRow,
Review comment:
avro plugins is more for the extractors and decoders right. This is a
util method. And pinot-core cannot be made to depend on pinot-avro for this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on pull request #5934: Segment processing framework
npawar commented on pull request #5934: URL: https://github.com/apache/incubator-pinot/pull/5934#issuecomment-692401724 > Overall, I like that all the core components are interfaced out and easy to extend. I have put some comments. Some of them are questions or points that I would like to discuss. Addressed the comments. Added TODOs in code and description for those that will be handled in future PRs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] suvodeep-pyne commented on pull request #6015: [TE] Make data-sources configurable in ThirdEye Helm Chart
suvodeep-pyne commented on pull request #6015: URL: https://github.com/apache/incubator-pinot/pull/6015#issuecomment-692362285 Thanks @apucher ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] agrawaldevesh commented on issue #3513: DATETIMECONVERT udf does not work for customized timezone and bucket size > 1 day
agrawaldevesh commented on issue #3513: URL: https://github.com/apache/incubator-pinot/issues/3513#issuecomment-692362099 FWIW, @chenboat I added a preso compatible date_trunc that handles your ask: https://github.com/apache/incubator-pinot/pull/4740. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: [TE] Make data-sources configurable in ThirdEye Helm Chart (#6015)
This is an automated email from the ASF dual-hosted git repository.
apucher pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 14611ae [TE] Make data-sources configurable in ThirdEye Helm Chart
(#6015)
14611ae is described below
commit 14611aeb616bc2a09cf75fd4875b827ee80f259d
Author: Suvodeep Pyne
AuthorDate: Mon Sep 14 15:40:04 2020 -0700
[TE] Make data-sources configurable in ThirdEye Helm Chart (#6015)
Currently, the data-sources config file is hardcoded in the template.
This change allows user to replace the data-sources config in the config
directory thereby allowing the user to feed a custom configuration.
By default, TE expects the pinot-quickstart cluster to be the primary
data source.
---
.../thirdeye/config/data-sources/data-sources-config.yml | 12
kubernetes/helm/thirdeye/{ => config}/initdb.sql | 12
kubernetes/helm/thirdeye/install.sh| 5 -
kubernetes/helm/thirdeye/templates/common/configmap.yaml | 14 +-
4 files changed, 29 insertions(+), 14 deletions(-)
diff --git
a/kubernetes/helm/thirdeye/config/data-sources/data-sources-config.yml
b/kubernetes/helm/thirdeye/config/data-sources/data-sources-config.yml
new file mode 100644
index 000..c68f0ec
--- /dev/null
+++ b/kubernetes/helm/thirdeye/config/data-sources/data-sources-config.yml
@@ -0,0 +1,12 @@
+dataSourceConfigs:
+ # Default option is to run thirdeye with a pinot-quickstart cluster
+ - className:
org.apache.pinot.thirdeye.datasource.pinot.PinotThirdEyeDataSource
+properties:
+ zookeeperUrl: 'pinot-zookeeper:2181'
+ clusterName: 'pinot-quickstart'
+ controllerConnectionScheme: 'http'
+ controllerHost: 'pinot-controller'
+ controllerPort: 9000
+ cacheLoaderClassName:
org.apache.pinot.thirdeye.datasource.pinot.PinotControllerResponseCacheLoader
+metadataSourceConfigs:
+ - className:
org.apache.pinot.thirdeye.auto.onboard.AutoOnboardPinotMetadataSource
\ No newline at end of file
diff --git a/kubernetes/helm/thirdeye/initdb.sql
b/kubernetes/helm/thirdeye/config/initdb.sql
similarity index 96%
rename from kubernetes/helm/thirdeye/initdb.sql
rename to kubernetes/helm/thirdeye/config/initdb.sql
index 54475be..fd1c217 100644
--- a/kubernetes/helm/thirdeye/initdb.sql
+++ b/kubernetes/helm/thirdeye/config/initdb.sql
@@ -446,3 +446,15 @@ create table if not exists online_detection_data_index (
create index online_detection_data_id_idx ON
online_detection_data_index(base_id);
create index online_detection_data_dataset_idx ON
online_detection_data_index(dataset);
create index online_detection_data_metric_idx ON
online_detection_data_index(metric);
+
+create table if not exists anomaly_subscription_group_notification_index (
+base_id bigint(20) not null,
+anomaly_id bigint(20) not null,
+detection_config_id bigint(20) not null,
+create_time timestamp,
+update_time timestamp default current_timestamp,
+version int(10)
+) ENGINE=InnoDB;
+ALTER TABLE `anomaly_subscription_group_notification_index` ADD UNIQUE
`anomaly_subscription_group_notification_index`(anomaly_id);
+create index anomaly_subscription_group_anomaly_idx ON
anomaly_subscription_group_notification_index(anomaly_id);
+create index anomaly_subscription_group_detection_config_idx ON
anomaly_subscription_group_notification_index(anomaly_id)
diff --git a/kubernetes/helm/thirdeye/install.sh
b/kubernetes/helm/thirdeye/install.sh
index 83ebdf6..8489c77 100755
--- a/kubernetes/helm/thirdeye/install.sh
+++ b/kubernetes/helm/thirdeye/install.sh
@@ -9,4 +9,7 @@ helm dependency update
# Note:
# - initdb files must end with .sql
# - When injecting yaml config via terminal, the period ('.') must be escaped
and quoted
-helm install thirdeye . --set-file
mysql.initializationFiles."initdb\.sql"="./initdb.sql" $@
+helm install thirdeye . \
+ --set-file mysql.initializationFiles."initdb\.sql"="./config/initdb.sql" \
+ --set-file
thirdeye.config.dataSources="./config/data-sources/data-sources-config.yml" \
+ $@
diff --git a/kubernetes/helm/thirdeye/templates/common/configmap.yaml
b/kubernetes/helm/thirdeye/templates/common/configmap.yaml
index 5647d5d..7d77465 100644
--- a/kubernetes/helm/thirdeye/templates/common/configmap.yaml
+++ b/kubernetes/helm/thirdeye/templates/common/configmap.yaml
@@ -204,19 +204,7 @@ data:
seed: 4
namePrefixes: [System, Model, Campaign, Welcome, Pinot, ThirdEye]
- data-sources-config.yml: |-
-# Please put the mock data source as the first in this configuration.
-dataSourceConfigs:
- - className:
org.apache.pinot.thirdeye.datasource.pinot.PinotThirdEyeDataSource
-properties:
- zookeeperUrl: 'pinot-zookeeper:2181'
- clusterName: 'pinot-quickstart'
-
[GitHub] [incubator-pinot] apucher merged pull request #6015: [TE] Make data-sources configurable in ThirdEye Helm Chart
apucher merged pull request #6015: URL: https://github.com/apache/incubator-pinot/pull/6015 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] suvodeep-pyne opened a new pull request #6015: [TE] Make data-sources configurable in ThirdEye Helm Chart
suvodeep-pyne opened a new pull request #6015: URL: https://github.com/apache/incubator-pinot/pull/6015 Currently, the data-sources config file is hardcoded in the template. This change allows user to replace the data-sources config in the config directory thereby allowing the user to feed a custom configuration. By default, TE expects the pinot-quickstart cluster to be the primary data source. No code changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (dc77271 -> d9d43fb)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from dc77271 Feature/#5390 segment indexing reload status api (#5718) add d9d43fb [TE] Add ingress, traefik to helm chart (#5985) No new revisions were added by this update. Summary of changes: kubernetes/helm/thirdeye/templates/_helpers.tpl | 14 ++ kubernetes/helm/thirdeye/templates/frontend/ingress.yaml | 14 ++ kubernetes/helm/thirdeye/values.yaml | 4 3 files changed, 32 insertions(+) create mode 100644 kubernetes/helm/thirdeye/templates/frontend/ingress.yaml - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (d9d43fb -> f5606b4)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d9d43fb [TE] Add ingress, traefik to helm chart (#5985) add f5606b4 [TE] Added thirdeye-coordinator and thirdeye-dist modules (#6014) No new revisions were added by this update. Summary of changes: thirdeye/pom.xml | 17 thirdeye/thirdeye-coordinator/pom.xml | 43 ++ thirdeye/thirdeye-dist/pom.xml | 98 ++ thirdeye/thirdeye-dist/src/main/assembly/dist.xml | 63 ++ .../thirdeye-dist/src/main/bash/thirdeye.sh| 46 ++ thirdeye/thirdeye-pinot/pom.xml| 1 - 6 files changed, 253 insertions(+), 15 deletions(-) create mode 100644 thirdeye/thirdeye-coordinator/pom.xml create mode 100644 thirdeye/thirdeye-dist/pom.xml create mode 100644 thirdeye/thirdeye-dist/src/main/assembly/dist.xml copy docker/images/pinot-thirdeye/bin/start-thirdeye.sh => thirdeye/thirdeye-dist/src/main/bash/thirdeye.sh (64%) mode change 100755 => 100644 - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (dc77271 -> d9d43fb)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from dc77271 Feature/#5390 segment indexing reload status api (#5718) add d9d43fb [TE] Add ingress, traefik to helm chart (#5985) No new revisions were added by this update. Summary of changes: kubernetes/helm/thirdeye/templates/_helpers.tpl | 14 ++ kubernetes/helm/thirdeye/templates/frontend/ingress.yaml | 14 ++ kubernetes/helm/thirdeye/values.yaml | 4 3 files changed, 32 insertions(+) create mode 100644 kubernetes/helm/thirdeye/templates/frontend/ingress.yaml - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (d9d43fb -> f5606b4)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d9d43fb [TE] Add ingress, traefik to helm chart (#5985) add f5606b4 [TE] Added thirdeye-coordinator and thirdeye-dist modules (#6014) No new revisions were added by this update. Summary of changes: thirdeye/pom.xml | 17 thirdeye/thirdeye-coordinator/pom.xml | 43 ++ thirdeye/thirdeye-dist/pom.xml | 98 ++ thirdeye/thirdeye-dist/src/main/assembly/dist.xml | 63 ++ .../thirdeye-dist/src/main/bash/thirdeye.sh| 46 ++ thirdeye/thirdeye-pinot/pom.xml| 1 - 6 files changed, 253 insertions(+), 15 deletions(-) create mode 100644 thirdeye/thirdeye-coordinator/pom.xml create mode 100644 thirdeye/thirdeye-dist/pom.xml create mode 100644 thirdeye/thirdeye-dist/src/main/assembly/dist.xml copy docker/images/pinot-thirdeye/bin/start-thirdeye.sh => thirdeye/thirdeye-dist/src/main/bash/thirdeye.sh (64%) mode change 100755 => 100644 - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] apucher merged pull request #6014: [TE] Added thirdeye-coordinator and thirdeye-dist modules
apucher merged pull request #6014: URL: https://github.com/apache/incubator-pinot/pull/6014 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] apucher merged pull request #5985: [TE] Add ingress, traefik to helm chart
apucher merged pull request #5985: URL: https://github.com/apache/incubator-pinot/pull/5985 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6009: Adjust schema validation logic in AvroIngestionSchemaValidator
jackjlli commented on a change in pull request #6009:
URL: https://github.com/apache/incubator-pinot/pull/6009#discussion_r488192591
##
File path:
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java
##
@@ -111,36 +112,56 @@ private void validateSchemas() {
}
}
if (nonNullSchema != null) {
+ avroColumnSchema = nonNullSchema;
avroColumnType = nonNullSchema.getType();
}
}
- if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
-_dataTypeMismatch.addMismatchReason(String
-.format("The Pinot column: (%s: %s) doesn't match with the column
(%s: %s) in input %s schema.", columnName,
-fieldSpec.getDataType().name(), avroColumnSchema.getName(),
avroColumnType.toString(),
-getInputSchemaType()));
- }
-
if (fieldSpec.isSingleValueField()) {
+// check data type mismatch
+if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.", columnName,
+ fieldSpec.getDataType().name(), avroColumnName,
avroColumnType.toString(),
+ getInputSchemaType()));
+}
+// check single-value multi-value mismatch
if (avroColumnType.ordinal() <
org.apache.avro.Schema.Type.STRING.ordinal()) {
// the column is a complex structure
- _singleValueMultiValueFieldMismatch.addMismatchReason(String.format(
- "The Pinot column: %s is 'single-value' column but the column:
%s from input %s is 'multi-value' column.",
- columnName, avroColumnSchema.getName(), getInputSchemaType()));
+
getSingleValueMultiValueFieldMismatchResult().addMismatchReason(String
+ .format(
+ "The Pinot column: %s is 'single-value' column but the
column: %s from input %s is 'multi-value' column.",
+ columnName, avroColumnName, getInputSchemaType()));
}
} else {
+// check data type mismatch
Review comment:
Done.
##
File path:
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java
##
@@ -111,36 +112,56 @@ private void validateSchemas() {
}
}
if (nonNullSchema != null) {
+ avroColumnSchema = nonNullSchema;
avroColumnType = nonNullSchema.getType();
}
}
- if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
-_dataTypeMismatch.addMismatchReason(String
-.format("The Pinot column: (%s: %s) doesn't match with the column
(%s: %s) in input %s schema.", columnName,
-fieldSpec.getDataType().name(), avroColumnSchema.getName(),
avroColumnType.toString(),
-getInputSchemaType()));
- }
-
if (fieldSpec.isSingleValueField()) {
+// check data type mismatch
+if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.", columnName,
+ fieldSpec.getDataType().name(), avroColumnName,
avroColumnType.toString(),
+ getInputSchemaType()));
+}
+// check single-value multi-value mismatch
if (avroColumnType.ordinal() <
org.apache.avro.Schema.Type.STRING.ordinal()) {
// the column is a complex structure
- _singleValueMultiValueFieldMismatch.addMismatchReason(String.format(
- "The Pinot column: %s is 'single-value' column but the column:
%s from input %s is 'multi-value' column.",
- columnName, avroColumnSchema.getName(), getInputSchemaType()));
+
getSingleValueMultiValueFieldMismatchResult().addMismatchReason(String
+ .format(
+ "The Pinot column: %s is 'single-value' column but the
column: %s from input %s is 'multi-value' column.",
+ columnName, avroColumnName, getInputSchemaType()));
}
} else {
+// check data type mismatch
+FieldSpec.DataType dataTypeForMVColumn =
AvroUtils.extractFieldDataType(avroColumnField);
+if (fieldSpec.getDataType() != dataTypeForMVColumn) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.",
+ columnName, fieldSpec.getDataType().name(), avroColumnName,
dataTypeForMVColumn.name(),
+
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6009: Adjust schema validation logic in AvroIngestionSchemaValidator
jackjlli commented on a change in pull request #6009:
URL: https://github.com/apache/incubator-pinot/pull/6009#discussion_r488141168
##
File path:
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java
##
@@ -111,36 +112,56 @@ private void validateSchemas() {
}
}
if (nonNullSchema != null) {
+ avroColumnSchema = nonNullSchema;
avroColumnType = nonNullSchema.getType();
}
}
- if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
-_dataTypeMismatch.addMismatchReason(String
-.format("The Pinot column: (%s: %s) doesn't match with the column
(%s: %s) in input %s schema.", columnName,
-fieldSpec.getDataType().name(), avroColumnSchema.getName(),
avroColumnType.toString(),
-getInputSchemaType()));
- }
-
if (fieldSpec.isSingleValueField()) {
+// check data type mismatch
+if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.", columnName,
+ fieldSpec.getDataType().name(), avroColumnName,
avroColumnType.toString(),
Review comment:
This is called only once in `init()` method of
`SegmentIndexCreationDriver` class before any transformation function gets
called.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch fix-schema-validator updated (8dd0777 -> 8b4841e)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch fix-schema-validator in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 8dd0777 Address PR comments add 8b4841e Address PR comments This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (8dd0777) \ N -- N -- N refs/heads/fix-schema-validator (8b4841e) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../inputformat/avro/AvroIngestionSchemaValidator.java | 14 -- 1 file changed, 8 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6009: Adjust schema validation logic in AvroIngestionSchemaValidator
jackjlli commented on a change in pull request #6009:
URL: https://github.com/apache/incubator-pinot/pull/6009#discussion_r488185083
##
File path:
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java
##
@@ -111,36 +112,56 @@ private void validateSchemas() {
}
}
if (nonNullSchema != null) {
+ avroColumnSchema = nonNullSchema;
avroColumnType = nonNullSchema.getType();
}
}
- if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
-_dataTypeMismatch.addMismatchReason(String
-.format("The Pinot column: (%s: %s) doesn't match with the column
(%s: %s) in input %s schema.", columnName,
-fieldSpec.getDataType().name(), avroColumnSchema.getName(),
avroColumnType.toString(),
-getInputSchemaType()));
- }
-
if (fieldSpec.isSingleValueField()) {
+// check data type mismatch
+if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.", columnName,
+ fieldSpec.getDataType().name(), avroColumnName,
avroColumnType.toString(),
+ getInputSchemaType()));
+}
+// check single-value multi-value mismatch
if (avroColumnType.ordinal() <
org.apache.avro.Schema.Type.STRING.ordinal()) {
Review comment:
In terms of the data type, we can use
`AvroUtils.extractFieldDataType(avroColumnField);` to get the data type from
the deepest element of AVRO data, then we compare it with the one in Pinot
schema.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch fix-schema-validator updated (f09ca9a -> 8dd0777)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch fix-schema-validator in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from f09ca9a Reduce test file sizes add 8dd0777 Address PR comments No new revisions were added by this update. Summary of changes: .../avro/AvroIngestionSchemaValidator.java | 37 +++--- 1 file changed, 18 insertions(+), 19 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] suvodeep-pyne opened a new pull request #6014: [TE] Added thirdeye-coordinator and thirdeye-dist modules
suvodeep-pyne opened a new pull request #6014: URL: https://github.com/apache/incubator-pinot/pull/6014 [TE] Added thirdeye-coordinator and thirdeye-dist modules These are initial stubs created with a larger goal to modularize the codebase. - thirdeye-coordinator: This module is primarily aimed at housing the api server and other coordination tasks - thirdeye-dist: This will contain the thirdeye distribution and scripts to launch TE components This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6009: Adjust schema validation logic in AvroIngestionSchemaValidator
jackjlli commented on a change in pull request #6009:
URL: https://github.com/apache/incubator-pinot/pull/6009#discussion_r488141168
##
File path:
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java
##
@@ -111,36 +112,56 @@ private void validateSchemas() {
}
}
if (nonNullSchema != null) {
+ avroColumnSchema = nonNullSchema;
avroColumnType = nonNullSchema.getType();
}
}
- if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
-_dataTypeMismatch.addMismatchReason(String
-.format("The Pinot column: (%s: %s) doesn't match with the column
(%s: %s) in input %s schema.", columnName,
-fieldSpec.getDataType().name(), avroColumnSchema.getName(),
avroColumnType.toString(),
-getInputSchemaType()));
- }
-
if (fieldSpec.isSingleValueField()) {
+// check data type mismatch
+if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.", columnName,
+ fieldSpec.getDataType().name(), avroColumnName,
avroColumnType.toString(),
Review comment:
This is called only once when `init()` method in
`SegmentIndexCreationDriver` class before any transformation function gets
called.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on a change in pull request #6009: Adjust schema validation logic in AvroIngestionSchemaValidator
mayankshriv commented on a change in pull request #6009:
URL: https://github.com/apache/incubator-pinot/pull/6009#discussion_r488129792
##
File path:
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java
##
@@ -111,36 +112,56 @@ private void validateSchemas() {
}
}
if (nonNullSchema != null) {
+ avroColumnSchema = nonNullSchema;
avroColumnType = nonNullSchema.getType();
}
}
- if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
-_dataTypeMismatch.addMismatchReason(String
-.format("The Pinot column: (%s: %s) doesn't match with the column
(%s: %s) in input %s schema.", columnName,
-fieldSpec.getDataType().name(), avroColumnSchema.getName(),
avroColumnType.toString(),
-getInputSchemaType()));
- }
-
if (fieldSpec.isSingleValueField()) {
+// check data type mismatch
+if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.", columnName,
+ fieldSpec.getDataType().name(), avroColumnName,
avroColumnType.toString(),
Review comment:
Is this called before or after the ingestion transformation functions?
For example, an ingestion transform function could change the name/type of data
in input?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #6009: Adjust schema validation logic in AvroIngestionSchemaValidator
mcvsubbu commented on a change in pull request #6009:
URL: https://github.com/apache/incubator-pinot/pull/6009#discussion_r488078457
##
File path:
pinot-plugins/pinot-input-format/pinot-avro-base/src/main/java/org/apache/pinot/plugin/inputformat/avro/AvroIngestionSchemaValidator.java
##
@@ -111,36 +112,56 @@ private void validateSchemas() {
}
}
if (nonNullSchema != null) {
+ avroColumnSchema = nonNullSchema;
avroColumnType = nonNullSchema.getType();
}
}
- if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
-_dataTypeMismatch.addMismatchReason(String
-.format("The Pinot column: (%s: %s) doesn't match with the column
(%s: %s) in input %s schema.", columnName,
-fieldSpec.getDataType().name(), avroColumnSchema.getName(),
avroColumnType.toString(),
-getInputSchemaType()));
- }
-
if (fieldSpec.isSingleValueField()) {
+// check data type mismatch
+if
(!fieldSpec.getDataType().name().equalsIgnoreCase(avroColumnType.toString())) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.", columnName,
+ fieldSpec.getDataType().name(), avroColumnName,
avroColumnType.toString(),
+ getInputSchemaType()));
+}
+// check single-value multi-value mismatch
if (avroColumnType.ordinal() <
org.apache.avro.Schema.Type.STRING.ordinal()) {
// the column is a complex structure
- _singleValueMultiValueFieldMismatch.addMismatchReason(String.format(
- "The Pinot column: %s is 'single-value' column but the column:
%s from input %s is 'multi-value' column.",
- columnName, avroColumnSchema.getName(), getInputSchemaType()));
+
getSingleValueMultiValueFieldMismatchResult().addMismatchReason(String
+ .format(
+ "The Pinot column: %s is 'single-value' column but the
column: %s from input %s is 'multi-value' column.",
+ columnName, avroColumnName, getInputSchemaType()));
}
} else {
+// check data type mismatch
+FieldSpec.DataType dataTypeForMVColumn =
AvroUtils.extractFieldDataType(avroColumnField);
+if (fieldSpec.getDataType() != dataTypeForMVColumn) {
+ getDataTypeMismatchResult().addMismatchReason(String
+ .format("The Pinot column: (%s: %s) doesn't match with the
column (%s: %s) in input %s schema.",
+ columnName, fieldSpec.getDataType().name(), avroColumnName,
dataTypeForMVColumn.name(),
+ getInputSchemaType()));
+}
+// check single-value multi-value mismatch
if (avroColumnType.ordinal() >=
org.apache.avro.Schema.Type.STRING.ordinal()) {
// the column is a complex structure
- _singleValueMultiValueFieldMismatch.addMismatchReason(String.format(
- "The Pinot column: %s is 'multi-value' column but the column: %s
from input %s schema is 'single-value' column.",
- columnName, avroColumnSchema.getName(), getInputSchemaType()));
+
getSingleValueMultiValueFieldMismatchResult().addMismatchReason(String
Review comment:
Can you update the comment in line 134 to be clearer, or just remove the
comment, since the format string below says exactly what the mismatch is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg merged pull request #5718: Feature/#5390 segment indexing reload status api
kishoreg merged pull request #5718: URL: https://github.com/apache/incubator-pinot/pull/5718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Feature/#5390 segment indexing reload status api (#5718)
This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new dc77271 Feature/#5390 segment indexing reload status api (#5718) dc77271 is described below commit dc772712f3d52c0c91caa774c320f9ca54b855dd Author: Guruguha AuthorDate: Mon Sep 14 09:28:04 2020 -0700 Feature/#5390 segment indexing reload status api (#5718) * - initial feature push * Segment Reload API - added a new API endpoint for users to query segment reload status API - Table metadata from Server - added a new endpoint to fetch segment metadata - added helper classes and methods to fetch metadata from the server Tests - added test to server API to fetch metadata including indexing information * Code Refactor: - Moved status classes to logical places Logs - Added logging statements Tests - Added unit tests for Pinot Controller reload status and segment metadata API - Added unit tests for Pinot Server reload status and segment metadata API License Headers - Add license headers to files added to this feature * Code Review Changes - Updating code as per PR review comments * Adding comments to new classes and methods added as part of this feature Removing SegmentMetadataFetcher as it seemed redundant Refactoring code to save failed segment reload status API calls as part of response * Code refactor to accommodate PR comments Pinot codestyle corrections Moving ServerSegmentMetadataReader to util * Updating API definition for loadStatus * Code refactor to remove duplicate code * Code refactor to remove duplicate code * Code refactor as per PR comments * Updating segment loadStatus API to return long time than readable string * - Bug fix on the server API endpoint * Adding pretty print for server metadata response * Reverting incorrect filename refactoring Updating variable names to reflect their value type * removing unused method variable * Enabling Pretty print of server response. Code refactor to clean up lines that went beyond line length * Removing reload status API * Pretty print result * Fix test Co-authored-by: Neha Pawar --- .../api/resources/PinotSegmentRestletResource.java | 59 +- .../api/resources/ServerTableSizeReader.java | 49 ++--- .../controller/util/CompletionServiceHelper.java | 108 ++ .../util/ServerSegmentMetadataReader.java | 92 + .../pinot/controller/util/TableMetadataReader.java | 74 +++ .../controller/api/PinotSegmentsMetadataTest.java | 230 + .../immutable/ImmutableSegmentImpl.java| 4 + .../api/resources/SegmentMetadataFetcher.java | 135 .../pinot/server/api/resources/TablesResource.java | 22 +- .../pinot/server/api/TablesResourceTest.java | 55 - 10 files changed, 773 insertions(+), 55 deletions(-) diff --git a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java index cd622d2..876d3a9 100644 --- a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java +++ b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java @@ -19,14 +19,17 @@ package org.apache.pinot.controller.api.resources; import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonNode; import io.swagger.annotations.Api; import io.swagger.annotations.ApiOperation; import io.swagger.annotations.ApiParam; +import java.io.IOException; import java.util.ArrayList; import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.Executor; import javax.annotation.Nullable; import javax.inject.Inject; import javax.ws.rs.Consumes; @@ -42,16 +45,20 @@ import javax.ws.rs.WebApplicationException; import javax.ws.rs.core.MediaType; import javax.ws.rs.core.Response; import javax.ws.rs.core.Response.Status; +import org.apache.commons.httpclient.HttpConnectionManager; import org.apache.helix.ZNRecord; import org.apache.helix.store.zk.ZkHelixPropertyStore; +import org.apache.pinot.common.exception.InvalidConfigException; import org.apache.pinot.common.exception.TableNotFoundException; import org.apache.pinot.common.metadata.ZKMetadataProvider; import org.apache.pinot.common.metadata.segment.OfflineSegmentZKMetadata; import
[GitHub] [incubator-pinot] fx19880617 commented on issue #5977: Allow ServiceManager to install tables prior to listening on service ports or a healthy status
fx19880617 commented on issue #5977: URL: https://github.com/apache/incubator-pinot/issues/5977#issuecomment-691872386 One thing I can think of is to implement a tableBootstrapService, which implements: `org.apache.pinot.common.utils.ServiceStatusCallback`. Then you can enable it in ServiceManager. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
