[GitHub] [incubator-pinot] icefury71 opened a new pull request #5870: Adding integration test for null handling from realtime source
icefury71 opened a new pull request #5870: URL: https://github.com/apache/incubator-pinot/pull/5870 ## Description Adding an end-end integration test for creating a Pinot table with Kafka records including null values and running queries against this table (with the IS NOT NULL predicate). Related to #4230 ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) No Does this PR fix a zero-downtime upgrade introduced earlier? No Does this PR otherwise need attention when creating release notes? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Ruoyingw opened a new pull request #5869: Roundup decimal points when compressing AnomalyTimelinesView
Ruoyingw opened a new pull request #5869: URL: https://github.com/apache/incubator-pinot/pull/5869 ## Description Previously it is found that many saved prediction results in AnomalyTimelinesView contain up to 14 decimal points. This is a waste of storage with little added information on the values. This pr first round up the time series to 3 decimal points before compressing. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] cyrilou242 opened a new pull request #5868: Te bq connector
cyrilou242 opened a new pull request #5868: URL: https://github.com/apache/incubator-pinot/pull/5868 ## Description Added support of BigQuery as a datasource for ThirdEye. I followed https://thirdeye.readthedocs.io/en/latest/contribute_datasource.html. Design points: - The JDBC driver for BigQuery has to be downloaded and put into the maven project. Also, the driver requires specific version for some dependencies. A normal build is unchanged: `mvn install -DskipTests` and a build with bigquery is `mvn install -DskipTests -P bigquery` Also, I've added the same principle for `install.sh`: `./install.sh` has the same behavior as before, `./install.sh bigquery` downloads and install the driver, and compile with the bigquery profile. The install.sh loop can look a bit complex/over-engineered, but I am anticipating the fact that we will have other databases that require more ops or other profiles for specific dependencies. We already have another case for using CloudSQL on GCP, but it's not the purpose of this MR. To give you an idea, the result is `./install.sh bigquery cloudsql`. I am not a Java dev so I had no idea how to do this properly. Let me know if I you're thinking of a better way to do this. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) No Does this PR fix a zero-downtime upgrade introduced earlier? No Does this PR otherwise need attention when creating release notes? Things to consider: No ## Release Notes Added support for BigQuery as a datasource in ThirdEye. ## Documentation I've added a documentation page here `thirdeye/docs/bigquery.rst`, but I did not find how to build the doc so I'm not 100% sure it builds correctly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch add-metrics-jmx updated (5d77260 -> 425f58c)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch add-metrics-jmx in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 5d77260 Bump up dropwizard metrics version to 4.1.2 new 425f58c Bump up dropwizard metrics version to 4.1.2 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (5d77260) \ N -- N -- N refs/heads/add-metrics-jmx (425f58c) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Bump up dropwizard metrics version to 4.1.2
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch add-metrics-jmx
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 425f58c1bdbd3150f93c1275deada2d8eb814205
Author: Jack Li(Analytics Engineering)
AuthorDate: Wed Aug 12 17:24:14 2020 -0700
Bump up dropwizard metrics version to 4.1.2
---
.travis.yml| 58 +++---
.../pinot/common/metrics/AbstractMetrics.java | 2 +
...eporterMetricsRegistryRegistrationListener.java | 6 +++
.../apache/pinot/common/metrics/MetricsHelper.java | 5 ++
pom.xml| 2 +-
5 files changed, 43 insertions(+), 30 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index da44855..6f26380 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -35,35 +35,35 @@ stages:
jobs:
include:
-- name: "Integration Tests - OracleJDK 8"
- jdk: oraclejdk8
- script:
-- ./.travis/.travis_test.sh
- env:
-- RUN_INTEGRATION_TESTS=true
-- name: "Unit Tests - OracleJDK 8"
- jdk: oraclejdk8
- script:
-- ./.travis/.travis_test.sh
- env:
-- RUN_INTEGRATION_TESTS=false
-- name: "QuickStart - Java 8 & OpenJDK 14-15"
- jdk: oraclejdk8
- script:
-- ./.travis/.travis_quickstart.sh
-- jdk_switcher use openjdk8
-- ./.travis/.travis_install.sh
-- ./.travis/.travis_quickstart.sh
-- ./.travis/.travis_quickstart_openjdk.sh 14
-- ./.travis/.travis_quickstart_openjdk.sh 15
-- name: "QuickStart - OpenJDK 10-13"
- jdk: openjdk10
- script:
-- java -version
-- ./.travis/.travis_quickstart.sh
-- ./.travis/.travis_quickstart_openjdk.sh 11
-- ./.travis/.travis_quickstart_openjdk.sh 12
-- ./.travis/.travis_quickstart_openjdk.sh 13
+#- name: "Integration Tests - OracleJDK 8"
+# jdk: oraclejdk8
+# script:
+#- ./.travis/.travis_test.sh
+# env:
+#- RUN_INTEGRATION_TESTS=true
+#- name: "Unit Tests - OracleJDK 8"
+# jdk: oraclejdk8
+# script:
+#- ./.travis/.travis_test.sh
+# env:
+#- RUN_INTEGRATION_TESTS=false
+#- name: "QuickStart - Java 8 & OpenJDK 14-15"
+# jdk: oraclejdk8
+# script:
+#- ./.travis/.travis_quickstart.sh
+#- jdk_switcher use openjdk8
+#- ./.travis/.travis_install.sh
+#- ./.travis/.travis_quickstart.sh
+#- ./.travis/.travis_quickstart_openjdk.sh 14
+#- ./.travis/.travis_quickstart_openjdk.sh 15
+#- name: "QuickStart - OpenJDK 10-13"
+# jdk: openjdk10
+# script:
+#- java -version
+#- ./.travis/.travis_quickstart.sh
+#- ./.travis/.travis_quickstart_openjdk.sh 11
+#- ./.travis/.travis_quickstart_openjdk.sh 12
+#- ./.travis/.travis_quickstart_openjdk.sh 13
- stage: deploy
script:
- travis_wait 40 ./.travis/.travis_nightly_build.sh
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
index 5178fc1..cf40f69 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
@@ -376,6 +376,7 @@ public abstract class AbstractMetrics metricsRegistries = metricsRegistryMap.keySet();
+ LOGGER.info("Size of metricsRegistries: {}", metricsRegistries.size());
for (MetricsRegistry metricsRegistry : metricsRegistries) {
listener.onMetricsRegistryRegistered(metricsRegistry);
}
@@ -127,6 +129,7 @@ public class MetricsHelper {
* @return Meter
*/
public static Meter newMeter(MetricsRegistry registry, MetricName name,
String eventType, TimeUnit unit) {
+LOGGER.info("Adding new meter {}", name);
if (registry != null) {
return registry.newMeter(name, eventType, unit);
} else {
@@ -236,6 +239,7 @@ public class MetricsHelper {
* @return gauge
*/
public static Gauge newGauge(MetricsRegistry registry, MetricName
name, Gauge gauge) {
+LOGGER.info("Adding new gauge {}", name);
if (registry != null) {
return registry.newGauge(name, gauge);
} else {
@@ -247,6 +251,7 @@ public class MetricsHelper {
* Removes an existing metric
*/
public static void removeMetric(MetricsRegistry registry, MetricName name) {
+LOGGER.info("Removing new metric {}", name);
if (registry != null) {
registry.removeMetric(name);
} else {
diff --git a/pom.xml b/pom.xml
index c59189a..0b1be4d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -139,7 +139,7 @@
3.1
-3.2.3
+4.1.2
1.1.1.7
2.11.2
4.1.42.Final
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.a
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5738: HAVING and Post-Aggregation Support
Jackie-Jiang commented on issue #5738: URL: https://github.com/apache/incubator-pinot/issues/5738#issuecomment-674302948 Select post-aggregation support: #5867 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5867: [Post-Aggregation] Support post-aggregation in SELECT
Jackie-Jiang opened a new pull request #5867: URL: https://github.com/apache/incubator-pinot/pull/5867 ## Description Add `PostAggregationHandler` to handle the post-aggregation calculation Enhance `AggregationDataTableReducer` and `GroupByDataTableReducer` to support post-aggregation in SELECT This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (6dd54f8 -> 2b58bfb)
This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 6dd54f8 Fix the variable names for off-heap alloc configs (#5852) add 2b58bfb [TE] clean up legacy code (#5842) No new revisions were added by this update. Summary of changes: .../anomaly/alert/util/DataReportHelper.java | 276 - .../thirdeye/anomaly/alert/util/EmailHelper.java | 130 .../dashboard/resources/SummaryResource.java | 24 +- .../views/contributor/ContributionCell.java| 183 .../views/contributor/ContributionViewTable.java | 48 --- .../contributor/ContributionViewTableBuilder.java | 142 - .../views/contributor/ContributorViewHandler.java | 331 - .../views/contributor/ContributorViewRequest.java | 27 -- .../views/contributor/ContributorViewResponse.java | 109 --- .../thirdeye/detection/yaml/YamlResource.java | 8 +- .../content/BaseNotificationContent.java | 2 - .../pinot/thirdeye/detector/anomaly-report-v2.ftl | 136 - .../pinot/thirdeye/detector/anomaly-report.ftl | 150 -- .../thirdeye/detector/custom-anomaly-report.ftl| 96 -- .../detector/data-report-by-metric-dimension.ftl | 105 --- .../dashboard/handler/ContributorTest.java | 70 - 16 files changed, 20 insertions(+), 1817 deletions(-) delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/anomaly/alert/util/DataReportHelper.java delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/views/contributor/ContributionCell.java delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/views/contributor/ContributionViewTable.java delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/views/contributor/ContributionViewTableBuilder.java delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/views/contributor/ContributorViewHandler.java delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/views/contributor/ContributorViewRequest.java delete mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/views/contributor/ContributorViewResponse.java delete mode 100644 thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/anomaly-report-v2.ftl delete mode 100644 thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/anomaly-report.ftl delete mode 100644 thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/custom-anomaly-report.ftl delete mode 100644 thirdeye/thirdeye-pinot/src/main/resources/org/apache/pinot/thirdeye/detector/data-report-by-metric-dimension.ftl delete mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/dashboard/handler/ContributorTest.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai merged pull request #5842: [TE] clean up legacy code
akshayrai merged pull request #5842: URL: https://github.com/apache/incubator-pinot/pull/5842 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Bump up dropwizard metrics version to 4.1.2
This is an automated email from the ASF dual-hosted git repository.
jlli pushed a commit to branch add-metrics-jmx
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 5d772606e4c993e8b44d40a34fb2010c28695ac7
Author: Jack Li(Analytics Engineering)
AuthorDate: Wed Aug 12 17:24:14 2020 -0700
Bump up dropwizard metrics version to 4.1.2
---
.travis.yml| 58 +++---
.../pinot/common/metrics/AbstractMetrics.java | 2 +
...eporterMetricsRegistryRegistrationListener.java | 6 +++
.../apache/pinot/common/metrics/MetricsHelper.java | 5 ++
pom.xml| 4 +-
5 files changed, 44 insertions(+), 31 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index da44855..6f26380 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -35,35 +35,35 @@ stages:
jobs:
include:
-- name: "Integration Tests - OracleJDK 8"
- jdk: oraclejdk8
- script:
-- ./.travis/.travis_test.sh
- env:
-- RUN_INTEGRATION_TESTS=true
-- name: "Unit Tests - OracleJDK 8"
- jdk: oraclejdk8
- script:
-- ./.travis/.travis_test.sh
- env:
-- RUN_INTEGRATION_TESTS=false
-- name: "QuickStart - Java 8 & OpenJDK 14-15"
- jdk: oraclejdk8
- script:
-- ./.travis/.travis_quickstart.sh
-- jdk_switcher use openjdk8
-- ./.travis/.travis_install.sh
-- ./.travis/.travis_quickstart.sh
-- ./.travis/.travis_quickstart_openjdk.sh 14
-- ./.travis/.travis_quickstart_openjdk.sh 15
-- name: "QuickStart - OpenJDK 10-13"
- jdk: openjdk10
- script:
-- java -version
-- ./.travis/.travis_quickstart.sh
-- ./.travis/.travis_quickstart_openjdk.sh 11
-- ./.travis/.travis_quickstart_openjdk.sh 12
-- ./.travis/.travis_quickstart_openjdk.sh 13
+#- name: "Integration Tests - OracleJDK 8"
+# jdk: oraclejdk8
+# script:
+#- ./.travis/.travis_test.sh
+# env:
+#- RUN_INTEGRATION_TESTS=true
+#- name: "Unit Tests - OracleJDK 8"
+# jdk: oraclejdk8
+# script:
+#- ./.travis/.travis_test.sh
+# env:
+#- RUN_INTEGRATION_TESTS=false
+#- name: "QuickStart - Java 8 & OpenJDK 14-15"
+# jdk: oraclejdk8
+# script:
+#- ./.travis/.travis_quickstart.sh
+#- jdk_switcher use openjdk8
+#- ./.travis/.travis_install.sh
+#- ./.travis/.travis_quickstart.sh
+#- ./.travis/.travis_quickstart_openjdk.sh 14
+#- ./.travis/.travis_quickstart_openjdk.sh 15
+#- name: "QuickStart - OpenJDK 10-13"
+# jdk: openjdk10
+# script:
+#- java -version
+#- ./.travis/.travis_quickstart.sh
+#- ./.travis/.travis_quickstart_openjdk.sh 11
+#- ./.travis/.travis_quickstart_openjdk.sh 12
+#- ./.travis/.travis_quickstart_openjdk.sh 13
- stage: deploy
script:
- travis_wait 40 ./.travis/.travis_nightly_build.sh
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
index 5178fc1..cf40f69 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java
@@ -376,6 +376,7 @@ public abstract class AbstractMetrics metricsRegistries = metricsRegistryMap.keySet();
+ LOGGER.info("Size of metricsRegistries: {}", metricsRegistries.size());
for (MetricsRegistry metricsRegistry : metricsRegistries) {
listener.onMetricsRegistryRegistered(metricsRegistry);
}
@@ -127,6 +129,7 @@ public class MetricsHelper {
* @return Meter
*/
public static Meter newMeter(MetricsRegistry registry, MetricName name,
String eventType, TimeUnit unit) {
+LOGGER.info("Adding new meter {}", name);
if (registry != null) {
return registry.newMeter(name, eventType, unit);
} else {
@@ -236,6 +239,7 @@ public class MetricsHelper {
* @return gauge
*/
public static Gauge newGauge(MetricsRegistry registry, MetricName
name, Gauge gauge) {
+LOGGER.info("Adding new gauge {}", name);
if (registry != null) {
return registry.newGauge(name, gauge);
} else {
@@ -247,6 +251,7 @@ public class MetricsHelper {
* Removes an existing metric
*/
public static void removeMetric(MetricsRegistry registry, MetricName name) {
+LOGGER.info("Removing new metric {}", name);
if (registry != null) {
registry.removeMetric(name);
} else {
diff --git a/pom.xml b/pom.xml
index c59189a..b54f8a2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -139,7 +139,7 @@
3.1
-3.2.3
+4.1.2
1.1.1.7
2.11.2
4.1.42.Final
@@ -390,7 +390,7 @@
com.yammer.metrics
metrics-core
-2.2.0
+2.2.1
[incubator-pinot] branch add-metrics-jmx updated (f29be93 -> 5d77260)
This is an automated email from the ASF dual-hosted git repository. jlli pushed a change to branch add-metrics-jmx in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard f29be93 Bump up dropwizard metrics version to 4.1.2 new 5d77260 Bump up dropwizard metrics version to 4.1.2 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (f29be93) \ N -- N -- N refs/heads/add-metrics-jmx (5d77260) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/java/org/apache/pinot/common/metrics/AbstractMetrics.java | 2 ++ 1 file changed, 2 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5861: [TE] merge time series snapshot when merging anomalies
vincentchenjl commented on a change in pull request #5861:
URL: https://github.com/apache/incubator-pinot/pull/5861#discussion_r470889015
##
File path:
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/util/ThirdEyeUtilsTest.java
##
@@ -176,21 +179,51 @@ public void testExceptionToStringLimited() {
}
@Test
- public void testMergeAnomalyProperties() {
+ public void testMergeAnomalyProperties() throws Exception {
+final long now = System.currentTimeMillis();
+final long bucketSize = 300_000;
+final double parentVal = 1.0, childVal = 2.0;
+
Map parentProperties = new HashMap<>();
parentProperties.put("p1", "value1");
parentProperties.put("p2", "value2");
parentProperties.put("detectorComponentName", "rule1");
+parentProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now, bucketSize, 10,
parentVal).toJsonString());
+
Map childProperties = new HashMap<>();
childProperties.put("p1", "value3");
childProperties.put("c1", "value4");
childProperties.put("detectorComponentName", "rule2");
+childProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now + bucketSize, bucketSize, 10,
childVal).toJsonString());
Review comment:
Added it as a new test.
##
File path:
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/util/ThirdEyeUtilsTest.java
##
@@ -176,21 +179,51 @@ public void testExceptionToStringLimited() {
}
@Test
- public void testMergeAnomalyProperties() {
+ public void testMergeAnomalyProperties() throws Exception {
+final long now = System.currentTimeMillis();
+final long bucketSize = 300_000;
+final double parentVal = 1.0, childVal = 2.0;
+
Map parentProperties = new HashMap<>();
parentProperties.put("p1", "value1");
parentProperties.put("p2", "value2");
parentProperties.put("detectorComponentName", "rule1");
+parentProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now, bucketSize, 10,
parentVal).toJsonString());
+
Map childProperties = new HashMap<>();
childProperties.put("p1", "value3");
childProperties.put("c1", "value4");
childProperties.put("detectorComponentName", "rule2");
+childProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now + bucketSize, bucketSize, 10,
childVal).toJsonString());
ThirdEyeUtils.mergeAnomalyProperties(parentProperties, childProperties);
Assert.assertEquals(parentProperties.get("p1"), "value1");
Assert.assertEquals(parentProperties.get("p2"), "value2");
Assert.assertEquals(parentProperties.get("c1"), "value4");
Assert.assertEquals(parentProperties.get("detectorComponentName"),
"rule1,rule2");
+AnomalyTimelinesView merged =
AnomalyTimelinesView.fromJsonString(parentProperties.get("anomalyTimelinesView"));
+Assert.assertEquals(merged.getTimeBuckets().size(), 11);
+Assert.assertTrue(merged.getCurrentValues().get(0) - parentVal < 0.1);
+Assert.assertTrue(merged.getCurrentValues().get(1) - parentVal < 0.1);
+Assert.assertTrue(merged.getCurrentValues().get(10) - childVal < 0.1);
Review comment:
Good point.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jasonyanwenl commented on a change in pull request #5866: [TE] add anomaly detection as a service - new table for storing ad-hoc data
jasonyanwenl commented on a change in pull request #5866:
URL: https://github.com/apache/incubator-pinot/pull/5866#discussion_r470827343
##
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/detection/AnomalyDetectionResource.java
##
@@ -464,6 +420,38 @@ TaskDTO generateTaskConfig(DetectionConfigDTO
detectionConfigDTO,
return taskDTO;
}
+ long saveOnlineDetectionData(JsonNode payloadNode,
+ DatasetConfigDTO datasetConfigDTO, MetricConfigDTO metricConfigDTO)
+throws JsonProcessingException {
+JsonNode dataNode = payloadNode.get(DATA_FIELD);
+String timeColumnName = datasetConfigDTO.getTimeColumn();
+String datasetName = datasetConfigDTO.getDataset();
+String metricName = metricConfigDTO.getName();
+
Review comment:
Thanks Harley! Yeah, the formatting for the ad-hoc data will be
explained in the API doc. I am currently writing that. I will share it when I
finished it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh commented on a change in pull request #5865: [TE] anomaly search - hide child anomaly unless queried with anomaly ids
jihaozh commented on a change in pull request #5865:
URL: https://github.com/apache/incubator-pinot/pull/5865#discussion_r470810638
##
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/v2/anomalies/AnomalySearcher.java
##
@@ -70,7 +70,11 @@ public AnomalySearcher() {
* @return the result
*/
public Map search(AnomalySearchFilter searchFilter, int
limit, int offset) {
-Predicate predicate = Predicate.EQ("child", false);
+Predicate predicate = Predicate.NEQ("baseId", 0);
Review comment:
it will be tested by unit tests and staging before deploy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] harleyjj commented on a change in pull request #5866: [TE] add anomaly detection as a service - new table for storing ad-hoc data
harleyjj commented on a change in pull request #5866:
URL: https://github.com/apache/incubator-pinot/pull/5866#discussion_r470809557
##
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/detection/AnomalyDetectionResource.java
##
@@ -464,6 +420,38 @@ TaskDTO generateTaskConfig(DetectionConfigDTO
detectionConfigDTO,
return taskDTO;
}
+ long saveOnlineDetectionData(JsonNode payloadNode,
+ DatasetConfigDTO datasetConfigDTO, MetricConfigDTO metricConfigDTO)
+throws JsonProcessingException {
+JsonNode dataNode = payloadNode.get(DATA_FIELD);
+String timeColumnName = datasetConfigDTO.getTimeColumn();
+String datasetName = datasetConfigDTO.getDataset();
+String metricName = metricConfigDTO.getName();
+
Review comment:
@jasonyanwenl This code looks good. Are there docs that tell users the
formatting for adhoc data, or is that future work for Thirdeye?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch anomaly-searcher-child updated (d173efe -> 58469f2)
This is an automated email from the ASF dual-hosted git repository. jihao pushed a change to branch anomaly-searcher-child in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard d173efe [TE] anomaly search - hide child anomaly unless queried with anomaly ids add 58469f2 [TE] anomaly search - hide child anomaly unless queried with anomaly ids This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (d173efe) \ N -- N -- N refs/heads/anomaly-searcher-child (58469f2) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../v2/anomalies/AnomalySearcherTest.java | 22 +- 1 file changed, 21 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on issue #5863: Lack of documentation on derived metric expression
kishoreg commented on issue #5863: URL: https://github.com/apache/incubator-pinot/issues/5863#issuecomment-674220405 @suvodeep-pyne ^^ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Ruoyingw commented on a change in pull request #5861: [TE] merge time series snapshot when merging anomalies
Ruoyingw commented on a change in pull request #5861:
URL: https://github.com/apache/incubator-pinot/pull/5861#discussion_r470793110
##
File path:
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/util/ThirdEyeUtilsTest.java
##
@@ -176,21 +179,51 @@ public void testExceptionToStringLimited() {
}
@Test
- public void testMergeAnomalyProperties() {
+ public void testMergeAnomalyProperties() throws Exception {
+final long now = System.currentTimeMillis();
+final long bucketSize = 300_000;
+final double parentVal = 1.0, childVal = 2.0;
+
Map parentProperties = new HashMap<>();
parentProperties.put("p1", "value1");
parentProperties.put("p2", "value2");
parentProperties.put("detectorComponentName", "rule1");
+parentProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now, bucketSize, 10,
parentVal).toJsonString());
+
Map childProperties = new HashMap<>();
childProperties.put("p1", "value3");
childProperties.put("c1", "value4");
childProperties.put("detectorComponentName", "rule2");
+childProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now + bucketSize, bucketSize, 10,
childVal).toJsonString());
Review comment:
Can we add another unit test case where the child anomaly has a gap with
parent? say
`childProperties.put("anomalyTimelinesView", generateAnomalyTimelineView(now
+ bucketSize * 20, bucketSize, 10, childVal).toJsonString());`
##
File path:
thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/util/ThirdEyeUtilsTest.java
##
@@ -176,21 +179,51 @@ public void testExceptionToStringLimited() {
}
@Test
- public void testMergeAnomalyProperties() {
+ public void testMergeAnomalyProperties() throws Exception {
+final long now = System.currentTimeMillis();
+final long bucketSize = 300_000;
+final double parentVal = 1.0, childVal = 2.0;
+
Map parentProperties = new HashMap<>();
parentProperties.put("p1", "value1");
parentProperties.put("p2", "value2");
parentProperties.put("detectorComponentName", "rule1");
+parentProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now, bucketSize, 10,
parentVal).toJsonString());
+
Map childProperties = new HashMap<>();
childProperties.put("p1", "value3");
childProperties.put("c1", "value4");
childProperties.put("detectorComponentName", "rule2");
+childProperties.put("anomalyTimelinesView",
+generateAnomalyTimelineView(now + bucketSize, bucketSize, 10,
childVal).toJsonString());
ThirdEyeUtils.mergeAnomalyProperties(parentProperties, childProperties);
Assert.assertEquals(parentProperties.get("p1"), "value1");
Assert.assertEquals(parentProperties.get("p2"), "value2");
Assert.assertEquals(parentProperties.get("c1"), "value4");
Assert.assertEquals(parentProperties.get("detectorComponentName"),
"rule1,rule2");
+AnomalyTimelinesView merged =
AnomalyTimelinesView.fromJsonString(parentProperties.get("anomalyTimelinesView"));
+Assert.assertEquals(merged.getTimeBuckets().size(), 11);
+Assert.assertTrue(merged.getCurrentValues().get(0) - parentVal < 0.1);
+Assert.assertTrue(merged.getCurrentValues().get(1) - parentVal < 0.1);
+Assert.assertTrue(merged.getCurrentValues().get(10) - childVal < 0.1);
Review comment:
Probably better to compare with absolute difference: `abs(actual -
expected) < 0.1`. Or better:
we can use `Assert.assertEquals(actual, expected, delta)`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5864: Fixing the race condition that segment finished before ControllerLeaderLocator created.
fx19880617 commented on a change in pull request #5864:
URL: https://github.com/apache/incubator-pinot/pull/5864#discussion_r470792533
##
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
##
@@ -363,6 +363,7 @@ public void start()
LOGGER.info("Initializing server instance and registering state model
factory");
Utils.logVersions();
+ControllerLeaderLocator.create(_helixManager);
Review comment:
Here we only hold a reference to helixManager
since we set ServerInstance::start as a addPreConnectCallback, then the
server will always start after helixManager connected.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] sajjad-moradi commented on issue #5751: Support Serving Latest Offline Segments Immediately
sajjad-moradi commented on issue #5751: URL: https://github.com/apache/incubator-pinot/issues/5751#issuecomment-674202729 > @sajjad-moradi we would prefer to discuss table config items in a design doc. If you can update the design doc with the proposed config, then @ianvkoeppe can go for it. thanks. The doc is updated with config suggestion. Please review and leave your comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jasonyanwenl opened a new pull request #5866: [TE] add anomaly detection as a service - new table for storing ad-hoc data
jasonyanwenl opened a new pull request #5866: URL: https://github.com/apache/incubator-pinot/pull/5866 Phase 2 is separated into two parts. This PR is for the 1st part. The main change is that a new table is created for storing online ad-hoc data. The PR for Phase 1 can be found in #5769. Here are some other minor changes: * Remove the `cleanExistingState` method in the `AnomalyDetectionResource`. This is not necessary because each request is distinct now and the monitor job will periodically clean them (dataset/metric/detection configs, ad-hoc data). * Remove the suffix appended to metric as the metric name is allowed to be duplicate. The 2nd part of phase 2 is about creating a new endpoint to provide registering and CRUD operations on ad-hoc data. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] cecilynie commented on a change in pull request #5865: [TE] anomaly search - hide child anomaly unless queried with anomaly ids
cecilynie commented on a change in pull request #5865:
URL: https://github.com/apache/incubator-pinot/pull/5865#discussion_r470779043
##
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/v2/anomalies/AnomalySearcher.java
##
@@ -70,7 +70,11 @@ public AnomalySearcher() {
* @return the result
*/
public Map search(AnomalySearchFilter searchFilter, int
limit, int offset) {
-Predicate predicate = Predicate.EQ("child", false);
+Predicate predicate = Predicate.NEQ("baseId", 0);
Review comment:
Hi Jihao, is there a way for us to test the fix before waiting for the
next deployment? This is a bit critical which is impacting all entity
monitoring. Let's make sure it works when we deploy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh opened a new pull request #5865: [TE] anomaly search - hide child anomaly unless queried with anomaly ids
jihaozh opened a new pull request #5865: URL: https://github.com/apache/incubator-pinot/pull/5865 Previously, the child anomalies will be hidden from the anomalies page. However, if clicked from an entity alert email, it requires showing those anomalies. This PR fixes the issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch anomaly-searcher-child created (now d173efe)
This is an automated email from the ASF dual-hosted git repository. jihao pushed a change to branch anomaly-searcher-child in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at d173efe [TE] anomaly search - hide child anomaly unless queried with anomaly ids This branch includes the following new commits: new d173efe [TE] anomaly search - hide child anomaly unless queried with anomaly ids The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: [TE] anomaly search - hide child anomaly unless queried with anomaly ids
This is an automated email from the ASF dual-hosted git repository.
jihao pushed a commit to branch anomaly-searcher-child
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit d173efe30472153fe7d65c52c86926a077ea765f
Author: Jihao Zhang
AuthorDate: Fri Aug 14 10:50:22 2020 -0700
[TE] anomaly search - hide child anomaly unless queried with anomaly ids
---
.../thirdeye/dashboard/resources/v2/anomalies/AnomalySearcher.java | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/v2/anomalies/AnomalySearcher.java
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/v2/anomalies/AnomalySearcher.java
index eeacbad..0403a97 100644
---
a/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/v2/anomalies/AnomalySearcher.java
+++
b/thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/v2/anomalies/AnomalySearcher.java
@@ -70,7 +70,11 @@ public class AnomalySearcher {
* @return the result
*/
public Map search(AnomalySearchFilter searchFilter, int
limit, int offset) {
-Predicate predicate = Predicate.EQ("child", false);
+Predicate predicate = Predicate.NEQ("baseId", 0);
+// hide child anomaly unless queried with ids
+if (searchFilter.getAnomalyIds().isEmpty()) {
+ predicate = Predicate.EQ("child", false);
+}
if (searchFilter.getStartTime() != null) {
predicate = Predicate.AND(predicate, Predicate.LT("startTime",
searchFilter.getEndTime()));
}
@@ -152,5 +156,4 @@ public class AnomalySearcher {
}
return ImmutableMap.of("count", count, "limit", limit, "offset", offset,
"elements", results);
}
-
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5751: Support Serving Latest Offline Segments Immediately
mcvsubbu commented on issue #5751: URL: https://github.com/apache/incubator-pinot/issues/5751#issuecomment-674186656 @sajjad-moradi we would prefer to discuss table config items in a design doc. If you can update the design doc with the proposed config, then @ianvkoeppe can go for it. thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] sajjad-moradi commented on issue #5751: Support Serving Latest Offline Segments Immediately
sajjad-moradi commented on issue #5751: URL: https://github.com/apache/incubator-pinot/issues/5751#issuecomment-674175664 @ianvkoeppe The agreement on design doc is to go with approach 1 as the short term solution. Since this approach is very similar to what you provided in PR #5745, could you please reopen the PR and apply the new changes. Basically the boolean flag that you have defined in TableConfig need to be changed to a number indicating how many days (or hours) need to be deducted from the time boundary. Default value should be one for backward compatibility. Follow PR #4156 to make the changes on time boundary calculation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on pull request #5864: Fixing the race condition that segment finished before ControllerLeaderLocator created.
npawar commented on pull request #5864: URL: https://github.com/apache/incubator-pinot/pull/5864#issuecomment-674172740 This doesn't explain why we saw the first 4 segments complete successfully. Afaik, there was no restart between testTable__0__3 and testTable__0__4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on pull request #5864: Fixing the race condition that segment finished before ControllerLeaderLocator created.
mcvsubbu commented on pull request #5864: URL: https://github.com/apache/incubator-pinot/pull/5864#issuecomment-674146813 I think the right fix is to not field any state transitions until the start() method has completed. We can have the state transition call wait on a semaphore? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on a change in pull request #5864: Fixing the race condition that segment finished before ControllerLeaderLocator created.
mcvsubbu commented on a change in pull request #5864:
URL: https://github.com/apache/incubator-pinot/pull/5864#discussion_r470715315
##
File path:
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
##
@@ -363,6 +363,7 @@ public void start()
LOGGER.info("Initializing server instance and registering state model
factory");
Utils.logVersions();
+ControllerLeaderLocator.create(_helixManager);
Review comment:
ControllerLeaderLocator uses helixManager to get the controller resource
status, so you need to have the helix manager connected before any leader
locator functionality is invoked. So your use case will not work if the same
race condition happens again.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5864: Fixing the race condition that segment finished before ControllerLeaderLocator created.
fx19880617 opened a new pull request #5864:
URL: https://github.com/apache/incubator-pinot/pull/5864
## Description
Current Pinot server start ServerInstance before ControllerLeaderLocator
creation.
This is typically ok but in one extreme case that during realtime server
bootstrap, one segment is consumed to threshold and trying to persist, it will
use `ControllerLeaderLocator` in
`ServerSegmentCompletionProtocolHandler.createSegmentCompletionUrl(...)` which
may not be created yet.
Below is a sample logs:
```
2020/08/14 00:20:05.855 INFO [PluginManager] [main] Plugins root dir is
[/opt/pinot/plugins]
2020/08/14 00:20:05.869 INFO [PluginManager] [main] Loading all plugins.
Please use env variable 'plugins.include' to customize.
2020/08/14 00:20:05.869 INFO [PluginManager] [main] Trying to load plugin
[pinot-batch-ingestion-hadoop] from location
[/opt/pinot/plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop]
2020/08/14 00:20:05.870 INFO [PluginManager] [main] Successfully loaded
plugin [pinot-batch-ingestion-hadoop] from jar file
[/opt/pinot/plugins/pinot-batch-ingestion/pinot-batch-ingestion-hadoop/pinot-batch-ingestion-hadoop-0.5.0-SNAPSHOT-shaded.jar]
...
...
2020/08/14 00:20:06.199 INFO [ServerInstance] [main] Initializing server
instance
2020/08/14 00:20:06.199 INFO [ServerInstance] [main] Initializing server
metrics
2020/08/14 00:20:06.220 INFO [ServerInstance] [main] Initializing instance
data manager of class:
org.apache.pinot.server.starter.helix.HelixInstanceDataManager
2020/08/14 00:20:06.223 INFO [HelixInstanceDataManager] [main] Initializing
Helix instance data manager
2020/08/14 00:20:06.224 INFO [HelixInstanceDataManagerConfig] [main]
InstanceDataManagerConfig, key: datadir , value: /var/pinot/server/data/index
2020/08/14 00:20:06.225 INFO [HelixInstanceDataManagerConfig] [main]
InstanceDataManagerConfig, key: segmenttardir , value:
/var/pinot/server/data/segment
2020/08/14 00:20:06.225 INFO [HelixInstanceDataManagerConfig] [main]
InstanceDataManagerConfig, key: data.manager.class , value:
org.apache.pinot.server.starter.helix.HelixInstanceDataManager
...
2020/08/14 00:20:08.114 INFO [HelixTaskExecutor] [main] Scheduling message
553a8091-14b7-48a9-91b6-61c9c722112d:
testTable_REALTIME:testTable__0__4__20200813T2127Z, OFFLINE->CONSUMING
2020/08/14 00:20:08.115 INFO [HelixTaskExecutor] [main] Submit task:
553a8091-14b7-48a9-91b6-61c9c722112d to pool:
java.util.concurrent.ThreadPoolExecutor@48b0e701[Running, pool size = 1, active
threads = 1, queued tasks = 0, completed tasks = 0]
2020/08/14 00:20:08.203 INFO [HelixStateTransitionHandler]
[HelixTaskExecutor-message_handle_thread] Instance
Server_my-host-server-0.my-host-server-headless.my-host.svc.cluster.local_8098,
partition testTable__0__4__20200813T2127Z received state transition from
OFFLINE to CONSUMING on session 100e1f0f4d00022, message id:
553a8091-14b7-48a9-91b6-61c9c722112d
2020/08/14 00:20:08.203 INFO
[SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel]
[HelixTaskExecutor-message_handle_thread]
SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline() :
ZnRecord=553a8091-14b7-48a9-91b6-61c9c722112d, {CREATE_TIMESTAMP=1597364407619,
ClusterEventName=CurrentStateChange, EXECUTE_START_TIMESTAMP=1597364408128,
EXE_SESSION_ID=100e1f0f4d00022, FROM_STATE=OFFLINE,
MSG_ID=553a8091-14b7-48a9-91b6-61c9c722112d, MSG_STATE=read,
MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=testTable__0__4__20200813T2127Z,
READ_TIMESTAMP=1597364407995, RESOURCE_NAME=testTable_REALTIME,
RESOURCE_TAG=testTable_REALTIME,
SRC_NAME=my-host-controller-0.my-host-controller-headless.my-host.svc.cluster.local_9000,
SRC_SESSION_ID=100e1f0f4d0001e,
STATE_MODEL_DEF=SegmentOnlineOfflineStateModel,
STATE_MODEL_FACTORY_NAME=DEFAULT,
TGT_NAME=Server_my-host-server-0.my-host-server-headless.my-host.svc.cluster.local_8098,
TGT_SESSION_ID=100e1f0f4d00022, TO_STATE=CONSUMING}{}{}
, Stat=Stat {_version=0, _creationTime=1597364407632,
_modifiedTime=1597364407632, _ephemeralOwner=0}
2020/08/14 00:20:08.203 INFO
[SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel]
[HelixTaskExecutor-message_handle_thread]
SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline() :
ZnRecord=553a8091-14b7-48a9-91b6-61c9c722112d, {CREATE_TIMESTAMP=1597364407619,
ClusterEventName=CurrentStateChange, EXECUTE_START_TIMESTAMP=1597364408128,
EXE_SESSION_ID=100e1f0f4d00022, FROM_STATE=OFFLINE,
MSG_ID=553a8091-14b7-48a9-91b6-61c9c722112d, MSG_STATE=read,
MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=testTable__0__4__20200813T2127Z,
READ_TIMESTAMP=1597364407995, RESOURCE_NAME=testTable_REALTIME,
RESOURCE_TAG=testTable_REALTIME,
SRC_NAME=my-host-controller-0.my-host-controller-headless.my-host.svc.cluster.local_9000,
SRC_SESSION_ID=100e1f0f4d0001e,
STATE_MODEL_DEF=SegmentOnlineOfflineStateModel,
STATE_MODEL_FACTORY_NAME=DEFAULT,
[incubator-pinot] branch bugfixing_race_condition_segment_creation updated: Fixing the race condition that segment finished before ControllerLeaderLocator created.
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch bugfixing_race_condition_segment_creation
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to
refs/heads/bugfixing_race_condition_segment_creation by this push:
new 617f76b Fixing the race condition that segment finished before
ControllerLeaderLocator created.
617f76b is described below
commit 617f76bb326e8f8f4c0c67e835996c378008100c
Author: Xiang Fu
AuthorDate: Fri Aug 14 02:14:14 2020 -0700
Fixing the race condition that segment finished before
ControllerLeaderLocator created.
---
.../java/org/apache/pinot/server/starter/helix/HelixServerStarter.java | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
index 09b18ce..abe2275 100644
---
a/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
+++
b/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixServerStarter.java
@@ -363,6 +363,7 @@ public class HelixServerStarter implements ServiceStartable
{
LOGGER.info("Initializing server instance and registering state model
factory");
Utils.logVersions();
+ControllerLeaderLocator.create(_helixManager);
ServerSegmentCompletionProtocolHandler
.init(_serverConf.subset(SegmentCompletionProtocol.PREFIX_OF_CONFIG_OF_SEGMENT_UPLOADER));
ServerConf serverInstanceConfig =
DefaultHelixStarterServerConfig.getDefaultHelixServerConfig(_serverConf);
@@ -414,8 +415,6 @@ public class HelixServerStarter implements ServiceStartable
{
// Register the service status handler
registerServiceStatusHandler();
-ControllerLeaderLocator.create(_helixManager);
-
if (_serverConf
.getProperty(CONFIG_OF_STARTUP_ENABLE_SERVICE_STATUS_CHECK,
DEFAULT_STARTUP_ENABLE_SERVICE_STATUS_CHECK)) {
long endTimeMs = startTimeMs +
_serverConf.getProperty(CONFIG_OF_STARTUP_TIMEOUT_MS,
DEFAULT_STARTUP_TIMEOUT_MS);
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch bugfixing_race_condition_segment_creation created (now 1d4d47a)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch bugfixing_race_condition_segment_creation in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 1d4d47a Use AvroRecordExtractor in ParquetRecordReader (#5830) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] ritesh-commits opened a new issue #5863: Lack of documentation on derived metric expression
ritesh-commits opened a new issue #5863: URL: https://github.com/apache/incubator-pinot/issues/5863 - While configuring a new metric, there doesn't seem to be an option to provide derived metric expression which is required isSimpleRatioMetric. Because of this the SummaryResponse is always being computed using the runAdditiveCube algorithm. - runAdditiveCubeAlgorithm is having hardcoded MerticAggFunction as SUM, so even if we are configured another metric like AVG or COUNT, the default agg function is not being considered while constructing Bulk Requests in BaseCubePinotClient. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Fix the variable names for off-heap alloc configs (#5852)
This is an automated email from the ASF dual-hosted git repository.
snlee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 6dd54f8 Fix the variable names for off-heap alloc configs (#5852)
6dd54f8 is described below
commit 6dd54f8aee0678dca0c9fb6653a7f894158996ca
Author: Seunghyun Lee
AuthorDate: Fri Aug 14 00:00:30 2020 -0700
Fix the variable names for off-heap alloc configs (#5852)
---
.../data/manager/config/InstanceDataManagerConfig.java | 2 +-
.../manager/realtime/HLRealtimeSegmentDataManager.java | 6 +++---
.../manager/realtime/LLRealtimeSegmentDataManager.java | 4 ++--
.../core/segment/index/loader/IndexLoadingConfig.java| 16
.../starter/helix/HelixInstanceDataManagerConfig.java| 4 ++--
5 files changed, 16 insertions(+), 16 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
index 6b2bec9..6422140 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
@@ -47,7 +47,7 @@ public interface InstanceDataManagerConfig {
boolean isRealtimeOffHeapAllocation();
- boolean isDirectRealtimeOffheapAllocation();
+ boolean isDirectRealtimeOffHeapAllocation();
int getMaxParallelSegmentBuilds();
}
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
index 7fa51d1..74aea94 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
@@ -191,10 +191,10 @@ public class HLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
.setNoDictionaryColumns(indexLoadingConfig.getNoDictionaryColumns())
.setVarLengthDictionaryColumns(indexLoadingConfig.getVarLengthDictionaryColumns())
.setInvertedIndexColumns(invertedIndexColumns).setRealtimeSegmentZKMetadata(realtimeSegmentZKMetadata)
-
.setOffHeap(indexLoadingConfig.isRealtimeOffheapAllocation()).setMemoryManager(
+
.setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation()).setMemoryManager(
getMemoryManager(realtimeTableDataManager.getConsumerDir(),
_segmentName,
-indexLoadingConfig.isRealtimeOffheapAllocation(),
-indexLoadingConfig.isDirectRealtimeOffheapAllocation(),
serverMetrics))
+indexLoadingConfig.isRealtimeOffHeapAllocation(),
+indexLoadingConfig.isDirectRealtimeOffHeapAllocation(),
serverMetrics))
.setStatsHistory(realtimeTableDataManager.getStatsHistory())
.setNullHandlingEnabled(indexingConfig.isNullHandlingEnabled()).build();
_realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfig);
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 8944e15..fc43ba2 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -,7 +,7 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
segmentLogger =
LoggerFactory.getLogger(LLRealtimeSegmentDataManager.class.getName() + "_" +
_segmentNameStr);
_tableStreamName = _tableNameWithType + "_" + _streamTopic;
_memoryManager =
getMemoryManager(realtimeTableDataManager.getConsumerDir(), _segmentNameStr,
-indexLoadingConfig.isRealtimeOffheapAllocation(),
indexLoadingConfig.isDirectRealtimeOffheapAllocation(),
+indexLoadingConfig.isRealtimeOffHeapAllocation(),
indexLoadingConfig.isDirectRealtimeOffHeapAllocation(),
serverMetrics);
List sortedColumns = indexLoadingConfig.getSortedColumns();
@@ -1155,7 +1155,7 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
}
_segmentMaxRowCount = segmentMaxRowCount;
-_isOffHeap = indexLoadingConfig.isRealtimeOffheapAllocation();
+_isOffHeap = indexLoadingConfig.isRealtimeOffHeapAllocation();
_nullHandlingEnabled = indexingConfig.isNullHandlingEnabled();
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/IndexLoadingConfig.java
b/pin
[incubator-pinot] branch master updated: Fix the variable names for off-heap alloc configs (#5852)
This is an automated email from the ASF dual-hosted git repository.
snlee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 6dd54f8 Fix the variable names for off-heap alloc configs (#5852)
6dd54f8 is described below
commit 6dd54f8aee0678dca0c9fb6653a7f894158996ca
Author: Seunghyun Lee
AuthorDate: Fri Aug 14 00:00:30 2020 -0700
Fix the variable names for off-heap alloc configs (#5852)
---
.../data/manager/config/InstanceDataManagerConfig.java | 2 +-
.../manager/realtime/HLRealtimeSegmentDataManager.java | 6 +++---
.../manager/realtime/LLRealtimeSegmentDataManager.java | 4 ++--
.../core/segment/index/loader/IndexLoadingConfig.java| 16
.../starter/helix/HelixInstanceDataManagerConfig.java| 4 ++--
5 files changed, 16 insertions(+), 16 deletions(-)
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
index 6b2bec9..6422140 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/config/InstanceDataManagerConfig.java
@@ -47,7 +47,7 @@ public interface InstanceDataManagerConfig {
boolean isRealtimeOffHeapAllocation();
- boolean isDirectRealtimeOffheapAllocation();
+ boolean isDirectRealtimeOffHeapAllocation();
int getMaxParallelSegmentBuilds();
}
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
index 7fa51d1..74aea94 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/HLRealtimeSegmentDataManager.java
@@ -191,10 +191,10 @@ public class HLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
.setNoDictionaryColumns(indexLoadingConfig.getNoDictionaryColumns())
.setVarLengthDictionaryColumns(indexLoadingConfig.getVarLengthDictionaryColumns())
.setInvertedIndexColumns(invertedIndexColumns).setRealtimeSegmentZKMetadata(realtimeSegmentZKMetadata)
-
.setOffHeap(indexLoadingConfig.isRealtimeOffheapAllocation()).setMemoryManager(
+
.setOffHeap(indexLoadingConfig.isRealtimeOffHeapAllocation()).setMemoryManager(
getMemoryManager(realtimeTableDataManager.getConsumerDir(),
_segmentName,
-indexLoadingConfig.isRealtimeOffheapAllocation(),
-indexLoadingConfig.isDirectRealtimeOffheapAllocation(),
serverMetrics))
+indexLoadingConfig.isRealtimeOffHeapAllocation(),
+indexLoadingConfig.isDirectRealtimeOffHeapAllocation(),
serverMetrics))
.setStatsHistory(realtimeTableDataManager.getStatsHistory())
.setNullHandlingEnabled(indexingConfig.isNullHandlingEnabled()).build();
_realtimeSegment = new MutableSegmentImpl(realtimeSegmentConfig);
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
index 8944e15..fc43ba2 100644
---
a/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
+++
b/pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
@@ -,7 +,7 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
segmentLogger =
LoggerFactory.getLogger(LLRealtimeSegmentDataManager.class.getName() + "_" +
_segmentNameStr);
_tableStreamName = _tableNameWithType + "_" + _streamTopic;
_memoryManager =
getMemoryManager(realtimeTableDataManager.getConsumerDir(), _segmentNameStr,
-indexLoadingConfig.isRealtimeOffheapAllocation(),
indexLoadingConfig.isDirectRealtimeOffheapAllocation(),
+indexLoadingConfig.isRealtimeOffHeapAllocation(),
indexLoadingConfig.isDirectRealtimeOffHeapAllocation(),
serverMetrics);
List sortedColumns = indexLoadingConfig.getSortedColumns();
@@ -1155,7 +1155,7 @@ public class LLRealtimeSegmentDataManager extends
RealtimeSegmentDataManager {
}
_segmentMaxRowCount = segmentMaxRowCount;
-_isOffHeap = indexLoadingConfig.isRealtimeOffheapAllocation();
+_isOffHeap = indexLoadingConfig.isRealtimeOffHeapAllocation();
_nullHandlingEnabled = indexingConfig.isNullHandlingEnabled();
diff --git
a/pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/IndexLoadingConfig.java
b/pin
[GitHub] [incubator-pinot] snleee merged pull request #5852: Fix the variable names for off-heap alloc configs
snleee merged pull request #5852: URL: https://github.com/apache/incubator-pinot/pull/5852 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on pull request #5852: Fix the variable names for off-heap alloc configs
snleee commented on pull request #5852: URL: https://github.com/apache/incubator-pinot/pull/5852#issuecomment-673922031 @mcvsubbu I just found that some variable names are not consistent. ``` boolean isRealtimeOffHeapAllocation(); boolean isDirectRealtimeOffheapAllocation(); ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
