[GitHub] [incubator-pinot] kishoreg commented on issue #5850: Docker HEALTHCHECK guidance
kishoreg commented on issue #5850: URL: https://github.com/apache/incubator-pinot/issues/5850#issuecomment-672634078 providing a /health end point should be possible, we also need a swagger endpoint This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg edited a comment on issue #5850: Docker HEALTHCHECK guidance
kishoreg edited a comment on issue #5850: URL: https://github.com/apache/incubator-pinot/issues/5850#issuecomment-672634078 providing a /health end point should be possible, we also need a swagger resource This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] adriancole opened a new issue #5850: Docker HEALTHCHECK guidance
adriancole opened a new issue #5850: URL: https://github.com/apache/incubator-pinot/issues/5850 There is awesome flexibility we allow with anything from all-in-one until distributed deployment. I love it. This presents a healthcheck challenge as certain ordering constraints remain and we want staged things to work naturally. Consider the case of ServiceManager starting controller, server and broker. We'd want one "HEALTHCHECK" to check regardless of the process config.. ideally an HTTP path like /health can we do this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua edited a comment on pull request #5787: [Feature] - Spark Pinot Connector
mangrrua edited a comment on pull request #5787: URL: https://github.com/apache/incubator-pinot/pull/5787#issuecomment-672613922 > @fx19880617 @mangrrua anything pending on this? Seems no @kishoreg This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on pull request #5787: [Feature] - Spark Pinot Connector
mangrrua commented on pull request #5787: URL: https://github.com/apache/incubator-pinot/pull/5787#issuecomment-672613922 > @fx19880617 @mangrrua anything pending on this? Seems no This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mangrrua commented on a change in pull request #5787: [Feature] - Spark Pinot Connector
mangrrua commented on a change in pull request #5787:
URL: https://github.com/apache/incubator-pinot/pull/5787#discussion_r468554247
##
File path:
pinot-connectors/pinot-spark-connector/src/main/scala/org/apache/pinot/connector/spark/connector/PinotServerDataFetcher.scala
##
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.connector.spark.connector
+
+import java.util.{List => JList, Map => JMap}
+
+import com.yammer.metrics.core.MetricsRegistry
+import org.apache.helix.model.InstanceConfig
+import org.apache.pinot.common.metrics.BrokerMetrics
+import org.apache.pinot.common.request.BrokerRequest
+import org.apache.pinot.common.utils.DataTable
+import org.apache.pinot.connector.spark.connector.Constants.PinotTableTypes
+import org.apache.pinot.connector.spark.datasource.PinotDataSourceReadOptions
+import org.apache.pinot.connector.spark.exceptions.PinotException
+import org.apache.pinot.connector.spark.utils.Logging
+import org.apache.pinot.core.transport.{AsyncQueryResponse, QueryRouter,
ServerInstance}
+import org.apache.pinot.sql.parsers.CalciteSqlCompiler
+
+import scala.collection.JavaConverters._
+
+/**
+ * Fetch data from specified Pinot server.
+ */
+private[pinot] class PinotServerDataFetcher(
+partitionId: Int,
+pinotSplit: PinotSplit,
+dataSourceOptions: PinotDataSourceReadOptions)
+ extends Logging {
+ private val sqlCompiler = new CalciteSqlCompiler()
+ private val brokerId = "apache_spark"
Review comment:
I could not find any information about that for now. I'll look it in the
next release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #5718: Feature/#5390 segment indexing reload status api
kishoreg commented on a change in pull request #5718:
URL: https://github.com/apache/incubator-pinot/pull/5718#discussion_r469006145
##
File path:
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java
##
@@ -485,4 +503,80 @@ private void deleteSegmentsInternal(String
tableNameWithType, List segme
throw new ControllerApplicationException(LOGGER, e.getMessage(),
Response.Status.FORBIDDEN);
}
}
+
+ @GET
+ @Path("segments/{tableName}/reload-status")
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(value = "Status of segment reload", notes = "Status of segment
reload")
Review comment:
this can be done in another PR. Lets get this in and add the
optimizations as we need them. Million segments in a table is not a common use
case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5738: HAVING and Post-Aggregation Support
Jackie-Jiang commented on issue #5738: URL: https://github.com/apache/incubator-pinot/issues/5738#issuecomment-672571284 Scalar function type conversion support: #5849 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5849: [Post-aggregation] Support type conversion for all scalar functions
Jackie-Jiang opened a new pull request #5849: URL: https://github.com/apache/incubator-pinot/pull/5849 ## Description Support type conversion for all scalar functions. Parameter classes supported for type conversion using `PinotDataType`: - int/Integer - long/Long - float/Float - double/Double - String - byte[] Also handle function name with underscore in `FunctionRegistry`. Support type conversion for all features using the scalar function: - Compile time function in `CalciteSqlParser` - Record transform/filter during ingestion in `InbuiltFunctionEvaluator` - Transform during query execution in `ScalarTransformFunctionWrapper` Add `PostAggregationFunction` to handle post-aggregation calculation using the scalar function. Add `ArithmeticFunctions` for all the arithmetic scalar functions: - plus - minus - times - divide - mod - min - max - abs - ceil - floor - exp - ln - sqrt This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: DataGenerator to tolerate DATE_TIME and COMPLEX fields (#5848)
This is an automated email from the ASF dual-hosted git repository.
apucher pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new bb8b19e DataGenerator to tolerate DATE_TIME and COMPLEX fields (#5848)
bb8b19e is described below
commit bb8b19e48ebd20db1e09e9bcaf98f1031fdfa5de
Author: Alexander Pucher
AuthorDate: Tue Aug 11 20:08:33 2020 -0700
DataGenerator to tolerate DATE_TIME and COMPLEX fields (#5848)
---
.../org/apache/pinot/tools/admin/command/GenerateDataCommand.java| 5 +
1 file changed, 5 insertions(+)
diff --git
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
index e57a01a..fe3e182 100644
---
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
+++
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
@@ -205,6 +205,11 @@ public class GenerateDataCommand extends
AbstractBaseAdminCommand implements Com
timeUnits.put(col, tfs.getIncomingGranularitySpec().getTimeType());
break;
+// forward compatibility with pattern generator
+case DATE_TIME:
+case COMPLEX:
+ break;
+
default:
throw new RuntimeException("Invalid field type.");
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] apucher merged pull request #5848: DataGenerator should tolerate DATE_TIME and COMPLEX fields
apucher merged pull request #5848: URL: https://github.com/apache/incubator-pinot/pull/5848 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5820: Query execution error when grouping by multi-value dimension and non-dictionary encoded dimension
fx19880617 commented on issue #5820:
URL:
https://github.com/apache/incubator-pinot/issues/5820#issuecomment-672493742
Currently we are not supporting No-Dictionary multi-value column in groupby
right now.
The relative code path is in
`org.apache.pinot.core.query.aggregation.groupby.NoDictionaryMultiColumnGroupKeyGenerator`:
```
@Override
public void generateKeysForBlock(TransformBlock transformBlock, int[][]
groupKeys) {
// TODO: Support generating keys for multi-valued columns.
throw new UnsupportedOperationException("Operation not supported");
}
```
I'm looking into the support for this.
cc: @kishoreg @Jackie-Jiang
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] apucher commented on pull request #5848: DataGenerator should tolerate DATE_TIME and COMPLEX fields
apucher commented on pull request #5848: URL: https://github.com/apache/incubator-pinot/pull/5848#issuecomment-672448160 we may actually consider removing it. afaik there's no active use of it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] apucher commented on pull request #5848: DataGenerator should tolerate DATE_TIME and COMPLEX fields
apucher commented on pull request #5848: URL: https://github.com/apache/incubator-pinot/pull/5848#issuecomment-672447672 @Jackie-Jiang we already do when using the pattern data generator. the crappy old code is around for backwards compatibility only. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on pull request #5848: DataGenerator should tolerate DATE_TIME and COMPLEX fields
npawar commented on pull request #5848: URL: https://github.com/apache/incubator-pinot/pull/5848#issuecomment-672446619 it should be fine to ignore the field for now. Eventually we want to handle it just like TIME case This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #5848: DataGenerator should tolerate DATE_TIME and COMPLEX fields
Jackie-Jiang commented on pull request #5848: URL: https://github.com/apache/incubator-pinot/pull/5848#issuecomment-672440092 @npawar Please take a look and see whether we should support generating data for date time fields This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] apucher opened a new pull request #5848: tolerate DATE_TIME and COMPLEX fields
apucher opened a new pull request #5848: URL: https://github.com/apache/incubator-pinot/pull/5848 ## Description Fixes a forward compatibility issue with DATE_TIME fields in the data generator ## Upgrade Notes Does this PR prevent a zero down-time upgrade? **no** Does this PR fix a zero-downtime upgrade introduced earlier? **no** Does this PR otherwise need attention when creating release notes? Things to consider: **no** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (db48107 -> f4949e9)
This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from db48107 Adding controller healthcheck endpoint: /health (#5846) add f4949e9 [TE] Added reset application API +refactor (#5847) No new revisions were added by this update. Summary of changes: .../api/application/ApplicationResource.java | 132 + 1 file changed, 85 insertions(+), 47 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] akshayrai merged pull request #5847: [TE] Added /application/reset API +refactor
akshayrai merged pull request #5847: URL: https://github.com/apache/incubator-pinot/pull/5847 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch data-generator-fix-20200811 created (now 502a9d3)
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch data-generator-fix-20200811 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 502a9d3 tolerate DATE_TIME and COMPLEX fields This branch includes the following new commits: new 502a9d3 tolerate DATE_TIME and COMPLEX fields The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: tolerate DATE_TIME and COMPLEX fields
This is an automated email from the ASF dual-hosted git repository.
apucher pushed a commit to branch data-generator-fix-20200811
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 502a9d3a9d1fdbdf4dd39f19f54289d98f650499
Author: Alexander Pucher
AuthorDate: Tue Aug 11 18:09:44 2020 -0700
tolerate DATE_TIME and COMPLEX fields
---
.../org/apache/pinot/tools/admin/command/GenerateDataCommand.java| 5 +
1 file changed, 5 insertions(+)
diff --git
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
index e57a01a..fe3e182 100644
---
a/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
+++
b/pinot-tools/src/main/java/org/apache/pinot/tools/admin/command/GenerateDataCommand.java
@@ -205,6 +205,11 @@ public class GenerateDataCommand extends
AbstractBaseAdminCommand implements Com
timeUnits.put(col, tfs.getIncomingGranularitySpec().getTimeType());
break;
+// forward compatibility with pattern generator
+case DATE_TIME:
+case COMPLEX:
+ break;
+
default:
throw new RuntimeException("Invalid field type.");
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] suvodeep-pyne commented on pull request #5847: [TE] Added /application/reset API +refactor
suvodeep-pyne commented on pull request #5847: URL: https://github.com/apache/incubator-pinot/pull/5847#issuecomment-672416913 Thanks a lot @akshayrai ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] suvodeep-pyne opened a new pull request #5847: [TE] Added /application/reset API +refactor
suvodeep-pyne opened a new pull request #5847:
URL: https://github.com/apache/incubator-pinot/pull/5847
## Description
- Added /application/reset/{name}. Resets an application
- Reset Usage: Primarily for demo purposes. It cleans up all old alerts and
anomalies
- Refactored /delete API code. reuse of deleteDependents and getByName
methods
- Removed manual exception handling since Jersey throws 500 by default
No backward incompatible changes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] adriancole commented on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
adriancole commented on pull request #5818: URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-672397474 PS I had a bug in my comment https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670817753 now is fixed. I didn't notice the fork about JRE 9+ needing plugins also in classpath. since this script is independent of this change more an FYI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] adriancole edited a comment on pull request #5818: Makes Pinot work on Alpine Linux or Distroless + BusyBox
adriancole edited a comment on pull request #5818: URL: https://github.com/apache/incubator-pinot/pull/5818#issuecomment-670817753 fyi I'm currently inlining a script as when you consider docker knows the only image it will run on, the script doesn't need to do work like detect OS, JVM etc. In this case, I'm ok closing this out if it isn't working. ```sh #!/bin/sh # For java 9 and later version, we need to explicitly set Pinot Plugins directory into classpath. CLASSPATH=$(find lib plugins -type f|xargs echo|tr ' ' ':') # TODO: link to pinot issue why we need internal access exec java \ --add-opens java.base/jdk.internal.ref=ALL-UNNAMED \ -classpath $CLASSPATH \ $JAVA_OPTS \ -Dlog4j.configurationFile=etc/log4j2.properties \ -Dapp.name=pinot-admin \ -Dapp.pid=$$ \ -Dapp.repo=$PINOT_HOME/lib \ -Dapp.home=$PINOT_HOME \ -Dbasedir=$PINOT_HOME \ org.apache.pinot.tools.admin.PinotAdministrator \ "$@" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Adding controller healthcheck endpoint: /health (#5846)
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new db48107 Adding controller healthcheck endpoint: /health (#5846)
db48107 is described below
commit db48107e1e13e26f91da038ca8c2c2174b1ac5b3
Author: Xiang Fu
AuthorDate: Tue Aug 11 17:06:24 2020 -0700
Adding controller healthcheck endpoint: /health (#5846)
* Adding /health endpoint in pinot controller
* Adding controller health check endpoint
---
.../pinot/common/metrics/ControllerMeter.java | 2 ++
.../api/resources/PinotControllerHealthCheck.java | 26 +-
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
index 39ea8db..1e7fc46 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
@@ -26,6 +26,8 @@ import org.apache.pinot.common.Utils;
*/
public enum ControllerMeter implements AbstractMetrics.Meter {
HELIX_ZOOKEEPER_RECONNECTS("reconnects", true),
+ HEALTHCHECK_OK_CALLS("healthcheck", true),
+ HEALTHCHECK_BAD_CALLS("healthcheck", true),
CONTROLLER_INSTANCE_POST_ERROR("InstancePostError", true),
CONTROLLER_INSTANCE_DELETE_ERROR("InstanceDeleteError", true),
CONTROLLER_SEGMENT_UPLOAD_ERROR("SegmentUploadError", true),
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
index 7370085..d844989 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
@@ -26,8 +26,13 @@ import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
import org.apache.commons.lang.StringUtils;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.ServiceStatus;
import org.apache.pinot.controller.ControllerConf;
@@ -38,15 +43,34 @@ public class PinotControllerHealthCheck {
@Inject
ControllerConf controllerConf;
+ @Inject
+ private ControllerMetrics controllerMetrics;
+
@GET
@Path("pinot-controller/admin")
@ApiOperation(value = "Check controller health")
@ApiResponses(value = {@ApiResponse(code = 200, message = "Good")})
@Produces(MediaType.TEXT_PLAIN)
- public String checkHealth() {
+ public String checkHealthLegacy() {
if (StringUtils.isNotBlank(controllerConf.generateVipUrl())) {
return "GOOD";
}
return "";
}
+
+ @GET
+ @Path("health")
+ @ApiOperation(value = "Check controller health")
+ @ApiResponses(value = {@ApiResponse(code = 200, message = "Good")})
+ @Produces(MediaType.TEXT_PLAIN)
+ public String checkHealth() {
+ServiceStatus.Status status = ServiceStatus.getServiceStatus();
+if (status == ServiceStatus.Status.GOOD) {
+
controllerMetrics.addMeteredGlobalValue(ControllerMeter.HEALTHCHECK_OK_CALLS,
1);
+ return "OK";
+}
+
controllerMetrics.addMeteredGlobalValue(ControllerMeter.HEALTHCHECK_BAD_CALLS,
1);
+throw new WebApplicationException(String.format("Pinot broker status is
%s", status),
+Response.Status.SERVICE_UNAVAILABLE);
+ }
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #5846: Adding controller healthcheck endpoint: /health
fx19880617 merged pull request #5846: URL: https://github.com/apache/incubator-pinot/pull/5846 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 closed issue #5803: Controllers do not have a `/health` resource
fx19880617 closed issue #5803: URL: https://github.com/apache/incubator-pinot/issues/5803 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5846: Adding controller healthcheck endpoint: /health
fx19880617 opened a new pull request #5846: URL: https://github.com/apache/incubator-pinot/pull/5846 ## Description Adding controller health check endpoint: /health to be on par with other pinot components. Existing check: `/pinot-controller/admin` is still available without any change of the behavior. ## Release Notes Adding controller health check endpoint: `/health` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch adding-controller-healthcheck-endpoint created (now bd3e06f)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch adding-controller-healthcheck-endpoint in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at bd3e06f Adding controller health check endpoint This branch includes the following new commits: new 1820b59 Adding /health endpoint in pinot controller new bd3e06f Adding controller health check endpoint The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/02: Adding /health endpoint in pinot controller
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch adding-controller-healthcheck-endpoint
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit 1820b5967764a4175290eed641c685460256c728
Author: Xiang Fu
AuthorDate: Tue Aug 11 14:36:18 2020 -0700
Adding /health endpoint in pinot controller
---
.../controller/api/resources/PinotControllerHealthCheck.java | 12
1 file changed, 12 insertions(+)
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
index 7370085..5876b3e 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
@@ -43,6 +43,18 @@ public class PinotControllerHealthCheck {
@ApiOperation(value = "Check controller health")
@ApiResponses(value = {@ApiResponse(code = 200, message = "Good")})
@Produces(MediaType.TEXT_PLAIN)
+ public String checkHealthLegacy() {
+if (StringUtils.isNotBlank(controllerConf.generateVipUrl())) {
+ return "GOOD";
+}
+return "";
+ }
+
+ @GET
+ @Path("health")
+ @ApiOperation(value = "Check controller health")
+ @ApiResponses(value = {@ApiResponse(code = 200, message = "Good")})
+ @Produces(MediaType.TEXT_PLAIN)
public String checkHealth() {
if (StringUtils.isNotBlank(controllerConf.generateVipUrl())) {
return "GOOD";
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 02/02: Adding controller health check endpoint
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch adding-controller-healthcheck-endpoint
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit bd3e06f7fbd8ba56cd84bea8eacea89e7ea58c2b
Author: Xiang Fu
AuthorDate: Tue Aug 11 14:40:52 2020 -0700
Adding controller health check endpoint
---
.../apache/pinot/common/metrics/ControllerMeter.java | 2 ++
.../api/resources/PinotControllerHealthCheck.java | 18 +++---
2 files changed, 17 insertions(+), 3 deletions(-)
diff --git
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
index 39ea8db..1e7fc46 100644
---
a/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
+++
b/pinot-common/src/main/java/org/apache/pinot/common/metrics/ControllerMeter.java
@@ -26,6 +26,8 @@ import org.apache.pinot.common.Utils;
*/
public enum ControllerMeter implements AbstractMetrics.Meter {
HELIX_ZOOKEEPER_RECONNECTS("reconnects", true),
+ HEALTHCHECK_OK_CALLS("healthcheck", true),
+ HEALTHCHECK_BAD_CALLS("healthcheck", true),
CONTROLLER_INSTANCE_POST_ERROR("InstancePostError", true),
CONTROLLER_INSTANCE_DELETE_ERROR("InstanceDeleteError", true),
CONTROLLER_SEGMENT_UPLOAD_ERROR("SegmentUploadError", true),
diff --git
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
index 5876b3e..d844989 100644
---
a/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
+++
b/pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotControllerHealthCheck.java
@@ -26,8 +26,13 @@ import javax.inject.Inject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
+import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
import org.apache.commons.lang.StringUtils;
+import org.apache.pinot.common.metrics.ControllerMeter;
+import org.apache.pinot.common.metrics.ControllerMetrics;
+import org.apache.pinot.common.utils.ServiceStatus;
import org.apache.pinot.controller.ControllerConf;
@@ -38,6 +43,9 @@ public class PinotControllerHealthCheck {
@Inject
ControllerConf controllerConf;
+ @Inject
+ private ControllerMetrics controllerMetrics;
+
@GET
@Path("pinot-controller/admin")
@ApiOperation(value = "Check controller health")
@@ -56,9 +64,13 @@ public class PinotControllerHealthCheck {
@ApiResponses(value = {@ApiResponse(code = 200, message = "Good")})
@Produces(MediaType.TEXT_PLAIN)
public String checkHealth() {
-if (StringUtils.isNotBlank(controllerConf.generateVipUrl())) {
- return "GOOD";
+ServiceStatus.Status status = ServiceStatus.getServiceStatus();
+if (status == ServiceStatus.Status.GOOD) {
+
controllerMetrics.addMeteredGlobalValue(ControllerMeter.HEALTHCHECK_OK_CALLS,
1);
+ return "OK";
}
-return "";
+
controllerMetrics.addMeteredGlobalValue(ControllerMeter.HEALTHCHECK_BAD_CALLS,
1);
+throw new WebApplicationException(String.format("Pinot broker status is
%s", status),
+Response.Status.SERVICE_UNAVAILABLE);
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #5787: [Feature] - Spark Pinot Connector
kishoreg commented on pull request #5787: URL: https://github.com/apache/incubator-pinot/pull/5787#issuecomment-672287954 @fx19880617 @mangrrua anything pending on this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5803: Controllers do not have a `/health` resource
fx19880617 commented on issue #5803: URL: https://github.com/apache/incubator-pinot/issues/5803#issuecomment-672278520 Make sense, we will add `/health` to controller. Current health check for controller is using `/pinot-controller/admin` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5802: Running Pinot with an invalid zookeeper service name seems to error during clean-up (in addition to connection error)
fx19880617 commented on issue #5802: URL: https://github.com/apache/incubator-pinot/issues/5802#issuecomment-672277394 This will error out during start up as controller cannot connect to zookeeper for further processing. And you can safely ignore this helix cleanup as there is no connection to zookeeper hence no cleanup is required. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg merged pull request #5808: [TE] Dashboard Resource Refactor
kishoreg merged pull request #5808: URL: https://github.com/apache/incubator-pinot/pull/5808 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (1c754f8 -> 47323de)
This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 1c754f8 Added set-diff operators and changed distinctCountThetaSketch syntax (#5832) add 47323de [TE] Dashboard Resource Refactor (#5808) No new revisions were added by this update. Summary of changes: .../api/application/ApplicationResource.java | 3 +- .../api/detection/AnomalyDetectionResource.java| 1 - .../api/user/dashboard/UserDashboardResource.java | 3 +- .../dashboard/ThirdEyeDashboardApplication.java| 60 +- .../dashboard/resources/AdminResource.java | 33 +++- .../resources/AnomalyFlattenResource.java | 3 +- .../dashboard/resources/AutoOnboardResource.java | 3 +- .../dashboard/resources/CacheResource.java | 3 +- .../resources/CustomizedEventResource.java | 3 +- .../dashboard/resources/DashboardResource.java | 71 +++ .../dashboard/resources/DatasetConfigResource.java | 3 +- .../dashboard/resources/EntityManagerResource.java | 3 +- .../dashboard/resources/EntityMappingResource.java | 3 +- .../dashboard/resources/MetricConfigResource.java | 3 +- .../resources/OnboardDatasetMetricResource.java| 3 +- .../thirdeye/dashboard/resources/RootResource.java | 216 + .../dashboard/resources/SummaryResource.java | 9 +- .../dashboard/resources/ThirdEyeResource.java | 25 ++- .../dashboard/resources/v2/AnomaliesResource.java | 3 +- .../dashboard/resources/v2/ConfigResource.java | 3 +- .../dashboard/resources/v2/DataResource.java | 3 +- .../resources/v2/DetectionAlertResource.java | 3 +- .../resources/v2/alerts/AlertResource.java | 4 +- .../v2/anomalies/AnomalySearchResource.java| 4 +- .../dataset/DatasetAutoOnboardResource.java| 3 +- .../sql/resources/SqlDataSourceResource.java | 3 +- .../detection/DetectionConfigurationResource.java | 3 +- .../thirdeye/detection/DetectionResource.java | 16 +- .../thirdeye/detection/yaml/YamlResource.java | 3 +- 29 files changed, 364 insertions(+), 132 deletions(-) create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/dashboard/resources/RootResource.java - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] suvodeep-pyne commented on pull request #5808: [TE] Dashboard Resource Refactor
suvodeep-pyne commented on pull request #5808: URL: https://github.com/apache/incubator-pinot/pull/5808#issuecomment-672266277 You are very welcome @akshayrai ! And thanks a lot for the approval. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg closed issue #5835: Problem when streaming data from S3 - 'Bucket' not found
kishoreg closed issue #5835: URL: https://github.com/apache/incubator-pinot/issues/5835 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on issue #5843: Public API checker before release
yupeng9 commented on issue #5843: URL: https://github.com/apache/incubator-pinot/issues/5843#issuecomment-672199407 > Related Issue #4854 +1 to adding it to the testing tool This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv merged pull request #5832: Added set-diff operators and changed distinctCountThetaSketch syntax
mayankshriv merged pull request #5832: URL: https://github.com/apache/incubator-pinot/pull/5832 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (b268012 -> 1c754f8)
This is an automated email from the ASF dual-hosted git repository. mayanks pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from b268012 [TE] add anomaly detection as a service - Phase 1 (#5769) add 1c754f8 Added set-diff operators and changed distinctCountThetaSketch syntax (#5832) No new revisions were added by this update. Summary of changes: ...istinctCountThetaSketchAggregationFunction.java | 196 - .../BrokerRequestToQueryContextConverterTest.java | 6 +- .../queries/DistinctCountThetaSketchTest.java | 61 +-- .../tests/ThetaSketchIntegrationTest.java | 12 +- 4 files changed, 203 insertions(+), 72 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jihaozh merged pull request #5769: [TE] add anomaly detection as a service - Phase 1
jihaozh merged pull request #5769: URL: https://github.com/apache/incubator-pinot/pull/5769 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (9551062 -> b268012)
This is an automated email from the ASF dual-hosted git repository. jihao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 9551062 update Swagger (OpenAPI) configuration for HTTP+HTTPS (#5817) add b268012 [TE] add anomaly detection as a service - Phase 1 (#5769) No new revisions were added by this update. Summary of changes: .../anomaly/ThirdEyeAnomalyApplication.java| 8 +- .../anomaly/ThirdEyeAnomalyConfiguration.java | 9 + .../pinot/thirdeye/anomaly/task/TaskConstants.java | 3 +- .../pinot/thirdeye/anomaly/task/TaskDriver.java| 20 +- .../thirdeye/anomaly/task/TaskInfoFactory.java | 5 +- .../thirdeye/anomaly/task/TaskRunnerFactory.java | 3 + .../anomaly/utils/ThirdeyeMetricsUtil.java | 6 + .../api/detection/AnomalyDetectionResource.java| 746 + .../api/user/dashboard/UserDashboardResource.java | 2 +- .../dashboard/ThirdEyeDashboardApplication.java| 4 +- .../pinot/thirdeye/datalayer/bao/TaskManager.java | 8 + .../datalayer/bao/jdbc/TaskManagerImpl.java| 37 + .../thirdeye/datalayer/pojo/MetricConfigBean.java | 10 + .../online/OnlineThirdEyeDataSource.java | 199 ++ .../pinot/resultset/ThirdEyeResultSetUtils.java| 1 + .../detection/DetectionPipelineTaskInfo.java | 20 +- .../detection/DetectionPipelineTaskRunner.java | 25 +- .../main/resources/detection-config-template.yml | 24 + .../detection/AnomalyDetectionResourceTest.java| 232 +++ .../thirdeye/api/detection/payload-bad-custom.json | 4 + .../pinot/thirdeye/api/detection/payload-bad.json | 3 + .../api/detection/payload-good-custom.json | 4 + .../pinot/thirdeye/api/detection/payload-good.json | 4 + 23 files changed, 1363 insertions(+), 14 deletions(-) create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/detection/AnomalyDetectionResource.java create mode 100644 thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/datasource/online/OnlineThirdEyeDataSource.java create mode 100644 thirdeye/thirdeye-pinot/src/main/resources/detection-config-template.yml create mode 100644 thirdeye/thirdeye-pinot/src/test/java/org/apache/pinot/thirdeye/api/detection/AnomalyDetectionResourceTest.java create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/api/detection/payload-bad-custom.json create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/api/detection/payload-bad.json create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/api/detection/payload-good-custom.json create mode 100644 thirdeye/thirdeye-pinot/src/test/resources/org/apache/pinot/thirdeye/api/detection/payload-good.json - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jasonyanwenl commented on pull request #5769: [TE] add anomaly detection as a service - Phase 1
jasonyanwenl commented on pull request #5769: URL: https://github.com/apache/incubator-pinot/pull/5769#issuecomment-672181314 Thank you so much for the quick review. Really appreciate it! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] jasonyanwenl commented on a change in pull request #5769: [TE] add anomaly detection as a service - Phase 1
jasonyanwenl commented on a change in pull request #5769:
URL: https://github.com/apache/incubator-pinot/pull/5769#discussion_r467200428
##
File path:
thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/api/detection/AnomalyDetectionResource.java
##
@@ -0,0 +1,756 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.thirdeye.api.detection;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.fasterxml.jackson.databind.node.TextNode;
+import io.dropwizard.auth.Auth;
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import org.apache.pinot.thirdeye.anomaly.task.TaskConstants;
+import org.apache.pinot.thirdeye.api.Constants;
+import org.apache.pinot.thirdeye.api.user.dashboard.UserDashboardResource;
+import org.apache.pinot.thirdeye.auth.ThirdEyePrincipal;
+import org.apache.pinot.thirdeye.common.metric.MetricType;
+import org.apache.pinot.thirdeye.constant.MetricAggFunction;
+import org.apache.pinot.thirdeye.dashboard.resources.v2.pojo.AnomalySummary;
+import org.apache.pinot.thirdeye.datalayer.bao.*;
+import org.apache.pinot.thirdeye.datalayer.dto.DatasetConfigDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.DetectionConfigDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.MetricConfigDTO;
+import org.apache.pinot.thirdeye.datalayer.dto.TaskDTO;
+import org.apache.pinot.thirdeye.datalayer.util.Predicate;
+import org.apache.pinot.thirdeye.datasource.DAORegistry;
+import org.apache.pinot.thirdeye.datasource.ThirdEyeCacheRegistry;
+import org.apache.pinot.thirdeye.datasource.loader.AggregationLoader;
+import org.apache.pinot.thirdeye.datasource.loader.DefaultAggregationLoader;
+import org.apache.pinot.thirdeye.datasource.loader.DefaultTimeSeriesLoader;
+import org.apache.pinot.thirdeye.datasource.loader.TimeSeriesLoader;
+import org.apache.pinot.thirdeye.detection.*;
+import org.apache.pinot.thirdeye.detection.cache.builder.AnomaliesCacheBuilder;
+import
org.apache.pinot.thirdeye.detection.cache.builder.TimeSeriesCacheBuilder;
+import org.apache.pinot.thirdeye.detection.validators.DatasetConfigValidator;
+import org.apache.pinot.thirdeye.detection.validators.DetectionConfigValidator;
+import org.apache.pinot.thirdeye.detection.validators.MetricConfigValidator;
+import org.apache.pinot.thirdeye.detection.yaml.DetectionConfigTuner;
+import
org.apache.pinot.thirdeye.detection.yaml.translator.DetectionConfigTranslator;
+import org.apache.pinot.thirdeye.util.ThirdEyeUtils;
+import org.jfree.util.Log;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.yaml.snakeyaml.Yaml;
+import javax.ws.rs.*;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.core.UriBuilder;
+import java.util.*;
+import java.util.concurrent.TimeUnit;
+
+@Path("/anomaly-detection")
+@Api(tags = { Constants.DETECTION_TAG })
+public class AnomalyDetectionResource {
+ protected static final Logger LOG =
LoggerFactory.getLogger(AnomalyDetectionResource.class);
+
+ private static final String TEMPLATE_DETECTION_PATH =
"detection-config-template.yml";
+
+ /* Detection config fields */
+ private static final String DETECTION_YAML_FIELD = "detectionName";
+ private static final String DEFAULT_DETECTION_NAME = "online_detection";
+
+ /* Metric config fields */
+ private static final String DATASET_YAML_FIELD = "dataset";
+ private static final String DEFAULT_DATASET_NAME = "online_dataset";
+ private static final String DATATYPE_YAML_FIELD = "datatype";
+ private static final MetricType DEFAULT_DATA_TYPE = MetricType.DOUBLE;
+
+ /* Dataset config fields */
+ private static final String METRIC_YAML_FIELD = "metric";
+ private static final String DEFAULT_METRIC_NAME = "online_metric";
+ private static final String DEFAULT_METRIC_COLUMN = "metric";
+ private static final String TIME_COLUMN_YAML_FIELD
[GitHub] [incubator-pinot] KKcorps edited a comment on issue #5845: String date format - not working on data ingestion
KKcorps edited a comment on issue #5845: URL: https://github.com/apache/incubator-pinot/issues/5845#issuecomment-672149764 For future reference, add the following to the job spec if your time column is in `String` format ```yaml segmentNameGeneratorSpec: type: normalizedDate ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps commented on issue #5845: String date format - not working on data ingestion
KKcorps commented on issue #5845: URL: https://github.com/apache/incubator-pinot/issues/5845#issuecomment-672149764 For future reference, add the following to the job spec ```yaml segmentNameGeneratorSpec: type: normalizedDate ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] itzikiusa closed issue #5845: String date format - not working on data ingestion
itzikiusa closed issue #5845: URL: https://github.com/apache/incubator-pinot/issues/5845 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] itzikiusa commented on issue #5845: String date format - not working on data ingestion
itzikiusa commented on issue #5845: URL: https://github.com/apache/incubator-pinot/issues/5845#issuecomment-672147212 thanks @KKcorps can use the following config to solve the issue: ``` segmentNameGeneratorSpec: type: normalizedDate configs: segment.name.prefix: 'airlineStats_batch' exclude.sequence.id: true` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] itzikiusa opened a new issue #5845: String date format - not working on data ingestion
itzikiusa opened a new issue #5845:
URL: https://github.com/apache/incubator-pinot/issues/5845
Hi,
trying to ingest data with timestamp in a format of: '-MM-dd HH:mm:ss'
scheme:
```
{
"schemaName": "transcript",
"dimensionFieldSpecs": [
{
"name": "casino_id",
"dataType": "INT"
},
{
"name": "round_id",
"dataType": "STRING"
},
{
"name": "trans_id",
"dataType": "STRING"
},
{
"name": "provider_id",
"dataType": "INT"
},
{
"name": "game_id",
"dataType": "STRING"
}
],
"metricFieldSpecs": [
{
"name": "amount",
"dataType": "FLOAT"
},
{
"name": "bonus_money",
"dataType": "FLOAT"
},
{
"name": "real_money",
"dataType": "FLOAT"
}
],
"dateTimeFieldSpecs": [{
"name": "timestamp",
"dataType": "STRING",
"format" : "1:SECONDS:SIMPLE_DATE_FORMAT:-MM-dd HH:mm:ss",
"granularity": "1:DAYS"
}]
}
and getting this error:
Got exception to kick off standalone data ingestion job -
java.lang.RuntimeException: Caught exception during running -
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:137)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:110)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:123)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:150)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:162)
[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
Caused by: java.lang.IllegalArgumentException: Illegal character in scheme
name at index 10: transcript_OFFLINE_2020-08-08 03:01:30_2020-08-08
03:01:30_0.tar.gz
at java.net.URI.create(URI.java:852) ~[?:1.8.0_252]
at java.net.URI.resolve(URI.java:1036) ~[?:1.8.0_252]
at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.run(SegmentGenerationJobRunner.java:204)
~[pinot-batch-ingestion-standalone-0.5.0-SNAPSHOT-shaded.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:135)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
... 4 more
Caused by: java.net.URISyntaxException: Illegal character in scheme name at
index 10: transcript_OFFLINE_2020-08-08 03:01:30_2020-08-08 03:01:30_0.tar.gz
at java.net.URI$Parser.fail(URI.java:2848) ~[?:1.8.0_252]
at java.net.URI$Parser.checkChars(URI.java:3021) ~[?:1.8.0_252]
at java.net.URI$Parser.parse(URI.java:3048) ~[?:1.8.0_252]
at java.net.URI.(URI.java:588) ~[?:1.8.0_252]
at java.net.URI.create(URI.java:850) ~[?:1.8.0_252]
at java.net.URI.resolve(URI.java:1036) ~[?:1.8.0_252]
at
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.run(SegmentGenerationJobRunner.java:204)
~[pinot-batch-ingestion-standalone-0.5.0-SNAPSHOT-shaded.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:135)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
... 4 more
Exception caught:
java.lang.RuntimeException: Caught exception during running -
org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:137)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:110)
~[pinot-all-0.5.0-SNAPSHOT-jar-with-dependencies.jar:0.5.0-SNAPSHOT-d8264c144f9655ebfa90059b5c9f74a007363f54]
at
org.apache.pinot.tools.admin.command.La
[GitHub] [incubator-pinot] itzikiusa commented on issue #5835: Problem when streaming data from S3 - 'Bucket' not found
itzikiusa commented on issue #5835: URL: https://github.com/apache/incubator-pinot/issues/5835#issuecomment-672082911 fixed on merged version This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #5843: Public API checker before release
mcvsubbu commented on issue #5843: URL: https://github.com/apache/incubator-pinot/issues/5843#issuecomment-672053619 Related Issue #4854 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg closed issue #5814: Pinot's swagger pages does not work on HTTPS-only environments
kishoreg closed issue #5814: URL: https://github.com/apache/incubator-pinot/issues/5814 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #4854: Compatibility regression testing tool
mcvsubbu commented on issue #4854: URL: https://github.com/apache/incubator-pinot/issues/4854#issuecomment-672053230 Related issue #5843 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (d8264c1 -> 9551062)
This is an automated email from the ASF dual-hosted git repository. kishoreg pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from d8264c1 Fix data ingestion from Amazon S3 bucket (#5836) add 9551062 update Swagger (OpenAPI) configuration for HTTP+HTTPS (#5817) No new revisions were added by this update. Summary of changes: LICENSE-binary | 2 +- .../pinot/broker/broker/BrokerAdminApiApplication.java | 4 ++-- .../org/apache/pinot/controller/ControllerConf.java | 9 - .../org/apache/pinot/controller/ControllerStarter.java | 2 +- .../controller/api/ControllerAdminApiApplication.java | 14 +- .../pinot/controller/util/ListenerConfigUtil.java | 17 - .../pinot/controller/util/ListenerConfigUtilTest.java | 6 -- .../pinot/server/starter/helix/AdminApiApplication.java | 4 ++-- .../service/PinotServiceManagerAdminApiApplication.java | 4 ++-- pom.xml | 2 +- 10 files changed, 14 insertions(+), 50 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg merged pull request #5817: update Swagger (OpenAPI) configuration for HTTP+HTTPS
kishoreg merged pull request #5817: URL: https://github.com/apache/incubator-pinot/pull/5817 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 merged pull request #5836: Fix data ingestion from Amazon S3 bucket
fx19880617 merged pull request #5836: URL: https://github.com/apache/incubator-pinot/pull/5836 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated (c04b8b3 -> d8264c1)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from c04b8b3 Move quickstart tests to use Java 14 release (#5844) add d8264c1 Fix data ingestion from Amazon S3 bucket (#5836) No new revisions were added by this update. Summary of changes: .../batch/standalone/SegmentGenerationJobRunner.java| 2 +- .../java/org/apache/pinot/plugin/filesystem/S3PinotFS.java | 13 ++--- .../org/apache/pinot/plugin/filesystem/S3PinotFSTest.java | 13 + 3 files changed, 20 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5704: Support toWeek/toMonth/toYear in datetimeConvertUdf
fx19880617 commented on issue #5704: URL: https://github.com/apache/incubator-pinot/issues/5704#issuecomment-671852467 We may also want to support this as granular in `datetimeConvertUdf`, so that we can take the pushdown from presto side for function: `date_trunc`(https://prestodb.io/docs/current/functions/datetime.html). |Unit | Example Truncated Value| |-- | --| |second | 2001-08-22 03:04:05.000 |minute | 2001-08-22 03:04:00.000 |hour | 2001-08-22 03:00:00.000 |day | 2001-08-22 00:00:00.000 |week | 2001-08-20 00:00:00.000 |month | 2001-08-01 00:00:00.000 |quarter | 2001-07-01 00:00:00.000 |year | 2001-01-01 00:00:00.000 ``` The above examples use the timestamp 2001-08-22 03:04:05.321 as the input. date_trunc(unit, x) → [same as input] Returns x truncated to unit. ``` Currently we are missing `week`/`month`/`quarter`/`year`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5836: Fix data ingestion from Amazon S3 bucket
KKcorps commented on a change in pull request #5836:
URL: https://github.com/apache/incubator-pinot/pull/5836#discussion_r468406304
##
File path:
pinot-plugins/pinot-file-system/pinot-s3/src/test/java/org/apache/pinot/plugin/filesystem/S3PinotFSTest.java
##
@@ -28,12 +28,16 @@
import java.util.Arrays;
import java.util.List;
import org.apache.commons.io.IOUtils;
+import org.apache.pinot.spi.env.PinotConfiguration;
Review comment:
Removed unused imports.
##
File path:
pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java
##
@@ -391,7 +393,11 @@ public long length(URI fileUri)
listObjectsV2Response.contents().stream().forEach(object -> {
//Only add files and not directories
if (!object.key().equals(fileUri.getPath()) &&
!object.key().endsWith(DELIMITER)) {
- builder.add(object.key());
+ String fileKey = object.key();
+ if(fileKey.startsWith(DELIMITER)){
Review comment:
does it look fine now?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
svn commit: r40941 - /release/incubator/pinot/KEYS
Author: tingchen Date: Tue Aug 11 07:08:15 2020 New Revision: 40941 Log: Update Pinot KEYS file for 1AAD3FDA Modified: release/incubator/pinot/KEYS Modified: release/incubator/pinot/KEYS == --- release/incubator/pinot/KEYS (original) +++ release/incubator/pinot/KEYS Tue Aug 11 07:08:15 2020 @@ -216,7 +216,7 @@ sub 4096R/6B55D0C6 2020-08-04 sig 1AAD3FDA 2020-08-04 tingchen (Apache org) -BEGIN PGP PUBLIC KEY BLOCK- -Version: GnuPG v1 +Version: GnuPG v2.0.22 (GNU/Linux) mQINBF8plTEBEAC7vlkP+ju0tUo+yhq9vcRSe/bjmVIYTf5QXCuR5nMAedmqk98+ LETrqSd4ikLByr5TpjE9FBfp9XVWEZlLriiVhbN6fyQsCvbQBRLC3G48n0uUtrbw - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
