[GitHub] [incubator-pinot] apucher opened a new pull request #6041: remove default javaagent opts in generator.sh script to avoid javaagent port colission

2020-09-21 Thread GitBox
apucher opened a new pull request #6041: URL: https://github.com/apache/incubator-pinot/pull/6041 ## Description This bugfix removes the default javaagent opts in generator.sh script (**only**) to avoid javaagent port collisions when running pinot-admin from a container that already

[GitHub] [incubator-pinot] jihaozh merged pull request #6026: [TE] Creating a thirdeye-dashboard module to host the dashboard server

2020-09-21 Thread GitBox
jihaozh merged pull request #6026: URL: https://github.com/apache/incubator-pinot/pull/6026 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[incubator-pinot] branch master updated (274b4c2 -> b5e67c9)

2020-09-21 Thread jihao
This is an automated email from the ASF dual-hosted git repository. jihao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 274b4c2 remove default javaagent opts in generator.sh script to avoid javaagent port colission (#6041)

[incubator-pinot] 01/01: remove default javaagent opts in generator.sh script to avoid javaagent port colission

2020-09-21 Thread apucher
This is an automated email from the ASF dual-hosted git repository. apucher pushed a commit to branch data-generator-javaagent-fix-20200921 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 918bb822bac9bff736c63088fb242d5c36fc8496 Author: Alexander Pucher AuthorDate

[incubator-pinot] branch data-generator-javaagent-fix-20200921 created (now 918bb82)

2020-09-21 Thread apucher
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch data-generator-javaagent-fix-20200921 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 918bb82 remove default javaagent opts in generator.sh script to avoid

[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6040: Remove the partition info from the consuming segment ZK metadata

2020-09-21 Thread GitBox
Jackie-Jiang commented on pull request #6040: URL: https://github.com/apache/incubator-pinot/pull/6040#issuecomment-696415308 @mcvsubbu The segment partition pruning for consuming segment will happen on server side instead of broker side. Yes there will be some overhead, and we need to

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #6042: Fix built-in virtual columns for immutable segment

2020-09-21 Thread GitBox
Jackie-Jiang opened a new pull request #6042: URL: https://github.com/apache/incubator-pinot/pull/6042 ## Description Fix the bug in `ImmutableSegmentLoader` where built-in virtual columns are not added to the schema in the segment metadata, which will cause wrong result when

[GitHub] [incubator-pinot] apucher merged pull request #6041: remove default javaagent opts in generator.sh script to avoid javaagent port colission

2020-09-21 Thread GitBox
apucher merged pull request #6041: URL: https://github.com/apache/incubator-pinot/pull/6041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[incubator-pinot] branch master updated (b65fe43 -> 274b4c2)

2020-09-21 Thread apucher
This is an automated email from the ASF dual-hosted git repository. apucher pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from b65fe43 [TE] fix labeler config mapping and timeout when fetching anomalies (#6036) add 274b4c2

[GitHub] [incubator-pinot] kishoreg commented on pull request #6037: Add list of allowed tables for emitting table level metrics

2020-09-21 Thread GitBox
kishoreg commented on pull request #6037: URL: https://github.com/apache/incubator-pinot/pull/6037#issuecomment-696321992 > > Aren’t we trying to address the limitations of monitoring systems in Pinot? > > I'm not aware that. Could you elaborate a bit more? what's exactly the plan

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #6043: Add IN_PARTITIONED_SUBQUERY support

2020-09-21 Thread GitBox
Jackie-Jiang opened a new pull request #6043: URL: https://github.com/apache/incubator-pinot/pull/6043 ## Description Add `IN_PARTITIONED_SUBQUERY` transform function to support `IDSET` aggregation function as the subquery on the server side. Because the subquery is solved on the

[GitHub] [incubator-pinot] mayankshriv opened a new pull request #6044: Support for multi-threaded Group By reducer for SQL.

2020-09-21 Thread GitBox
mayankshriv opened a new pull request #6044: URL: https://github.com/apache/incubator-pinot/pull/6044 The existing implementation of Broker reduce phase is single-threaded. For group-by queries where large response are being sent back from multiple servers, this could become a

[GitHub] [incubator-pinot] mayankshriv commented on issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance

2020-09-21 Thread GitBox
mayankshriv commented on issue #6028: URL: https://github.com/apache/incubator-pinot/issues/6028#issuecomment-696510899 @mr-agrwal Could you please try the changes in this PR and provide some feedback: https://github.com/apache/incubator-pinot/pull/6044

[GitHub] [incubator-pinot] kishoreg commented on issue #5942: Better Table config validation

2020-09-21 Thread GitBox
kishoreg commented on issue #5942: URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696214712 +1 to a utility to validate, we can add a validate endpoint in the controller This is an automated message

[GitHub] [incubator-pinot] sajjad-moradi commented on pull request #6037: Add list of allowed tables for emitting table level metrics

2020-09-21 Thread GitBox
sajjad-moradi commented on pull request #6037: URL: https://github.com/apache/incubator-pinot/pull/6037#issuecomment-696225516 > Aren’t we trying to address the limitations of monitoring systems in Pinot? I'm not aware that. Could you elaborate a bit more? what's exactly the plan

[GitHub] [incubator-pinot] apucher merged pull request #6041: remove default javaagent opts in generator.sh script to avoid javaagent port colission

2020-09-21 Thread GitBox
apucher merged pull request #6041: URL: https://github.com/apache/incubator-pinot/pull/6041 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-pinot] mayankshriv commented on pull request #6033: Add Broker Reduce Time Log

2020-09-21 Thread GitBox
mayankshriv commented on pull request #6033: URL: https://github.com/apache/incubator-pinot/pull/6033#issuecomment-696497353 @mr-agrwal @fx19880617 We do have metrics for each phase of query in the broker side (including scatter/gather, reduce, etc). Adding just the reduce part in the log

[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation

2020-09-21 Thread GitBox
icefury71 commented on issue #5942: URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696164071 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [incubator-pinot] adriancole commented on issue #5977: Allow ServiceManager to install tables prior to listening on service ports or a healthy status

2020-09-21 Thread GitBox
adriancole commented on issue #5977: URL: https://github.com/apache/incubator-pinot/issues/5977#issuecomment-696109515 ok I added a sketch for feedback. I will work on this again next week https://github.com/apache/incubator-pinot/pull/6039

[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #6020: Add Caching in Controller Broker API

2020-09-21 Thread GitBox
KKcorps commented on a change in pull request #6020: URL: https://github.com/apache/incubator-pinot/pull/6020#discussion_r491796919 ## File path: pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/listener/ClusterLiveInstanceChangeListener.java ## @@ -0,0

[GitHub] [incubator-pinot] mcvsubbu commented on issue #5942: Better Table config validation

2020-09-21 Thread GitBox
mcvsubbu commented on issue #5942: URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696207582 With all the changes proposed/added, it will be useful if a utility is also provided to validate an existing tableconfig. Many of them are tightening the (lax) rules from

[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6042: Fix built-in virtual columns for immutable segment

2020-09-21 Thread GitBox
Jackie-Jiang merged pull request #6042: URL: https://github.com/apache/incubator-pinot/pull/6042 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #6042: Fix built-in virtual columns for immutable segment

2020-09-21 Thread GitBox
Jackie-Jiang merged pull request #6042: URL: https://github.com/apache/incubator-pinot/pull/6042 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[incubator-pinot] branch master updated: Fix built-in virtual columns for immutable segment (#6042)

2020-09-21 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 67299cd Fix built-in virtual columns

[incubator-pinot] 01/01: [maven-release-plugin] prepare for next development iteration

2020-09-21 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch release-0.5.0-rc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit 7e7af116684950a8e5e86600d9c1a07bc5e0759c Author: TING CHEN AuthorDate: Wed Sep 2 17:49:37 2020

[incubator-pinot] branch release-0.5.0-rc created (now 67299cd)

2020-09-21 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a change to branch release-0.5.0-rc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 67299cd Fix built-in virtual columns for immutable segment (#6042) No new revisions were

[incubator-pinot] branch release-0.5.0-rc created (now 7e7af11)

2020-09-21 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a change to branch release-0.5.0-rc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at 7e7af11 [maven-release-plugin] prepare for next development iteration This branch includes

[GitHub] [incubator-pinot] mayankshriv edited a comment on issue #6028: Make BrokerReduceService.reduceOnDataTable Multi Threaded to increase aggregation performance

2020-09-21 Thread GitBox
mayankshriv edited a comment on issue #6028: URL: https://github.com/apache/incubator-pinot/issues/6028#issuecomment-696510899 @mr-agrwal Could you please try the changes in this PR and provide some feedback: https://github.com/apache/incubator-pinot/pull/6044

[GitHub] [incubator-pinot] kishoreg commented on pull request #6033: Add Broker Reduce Time Log

2020-09-21 Thread GitBox
kishoreg commented on pull request #6033: URL: https://github.com/apache/incubator-pinot/pull/6033#issuecomment-696501753 This is the one that uncovered the issue that broker was indeed taking a long time. This will probably help at LinkedIn as well to identify the usecases that can

[GitHub] [incubator-pinot] jihaozh merged pull request #6036: [TE] fix labeler config mapping and timeout when fetching anomalies

2020-09-21 Thread GitBox
jihaozh merged pull request #6036: URL: https://github.com/apache/incubator-pinot/pull/6036 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #6040: Remove the partition info from the consuming segment ZK metadata

2020-09-21 Thread GitBox
Jackie-Jiang commented on pull request #6040: URL: https://github.com/apache/incubator-pinot/pull/6040#issuecomment-696415308 @mcvsubbu The segment partition pruning for consuming segment will happen on server side instead of broker side. Yes there will be some overhead, and we need to

[GitHub] [incubator-pinot] jihaozh merged pull request #6026: [TE] Creating a thirdeye-dashboard module to host the dashboard server

2020-09-21 Thread GitBox
jihaozh merged pull request #6026: URL: https://github.com/apache/incubator-pinot/pull/6026 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #6040: Remove the partition info from the consuming segment ZK metadata

2020-09-21 Thread GitBox
Jackie-Jiang opened a new pull request #6040: URL: https://github.com/apache/incubator-pinot/pull/6040 ## Description For issue: #6029 Do not persist the partition info to the consuming segment ZK metadata because it might not reflect the correct segment partitioning when the data

[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation

2020-09-21 Thread GitBox
icefury71 commented on issue #5942: URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696260153 Updated ticket description with all the suggestions. This is an automated message from the Apache Git

[incubator-pinot] 01/01: Remove the partition info from the consuming segment ZK metadata

2020-09-21 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a commit to branch remove_consuming_partition_info in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git commit f83cbb569448d883250ae6d57d1070e80f5b3657 Author: Xiaotian (Jackie) Jiang AuthorDate:

[incubator-pinot] branch remove_consuming_partition_info updated (200bef6 -> f83cbb5)

2020-09-21 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch remove_consuming_partition_info in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard 200bef6 Remove the partition info from the consuming segment ZK metadata

[GitHub] [incubator-pinot] sajjad-moradi commented on pull request #6037: Add list of allowed tables for emitting table level metrics

2020-09-21 Thread GitBox
sajjad-moradi commented on pull request #6037: URL: https://github.com/apache/incubator-pinot/pull/6037#issuecomment-696225516 > Aren’t we trying to address the limitations of monitoring systems in Pinot? I'm not aware that. Could you elaborate a bit more? what's exactly the plan

[GitHub] [incubator-pinot] mcvsubbu commented on issue #5942: Better Table config validation

2020-09-21 Thread GitBox
mcvsubbu commented on issue #5942: URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696207582 With all the changes proposed/added, it will be useful if a utility is also provided to validate an existing tableconfig. Many of them are tightening the (lax) rules from

[GitHub] [incubator-pinot] kishoreg commented on issue #5942: Better Table config validation

2020-09-21 Thread GitBox
kishoreg commented on issue #5942: URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696214712 +1 to a utility to validate, we can add a validate endpoint in the controller This is an automated message

[GitHub] [incubator-pinot] icefury71 commented on issue #5942: Better Table config validation

2020-09-21 Thread GitBox
icefury71 commented on issue #5942: URL: https://github.com/apache/incubator-pinot/issues/5942#issuecomment-696164071 One more check: Currently retention does not work if pushType is null (for real-time table). Check for a case where pushType is null and retention is configured for a

[GitHub] [incubator-pinot] kishoreg commented on pull request #6037: Add list of allowed tables for emitting table level metrics

2020-09-21 Thread GitBox
kishoreg commented on pull request #6037: URL: https://github.com/apache/incubator-pinot/pull/6037#issuecomment-696321992 > > Aren’t we trying to address the limitations of monitoring systems in Pinot? > > I'm not aware that. Could you elaborate a bit more? what's exactly the plan

[GitHub] [incubator-pinot] jihaozh merged pull request #6036: [TE] fix labeler config mapping and timeout when fetching anomalies

2020-09-21 Thread GitBox
jihaozh merged pull request #6036: URL: https://github.com/apache/incubator-pinot/pull/6036 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[incubator-pinot] branch master updated (73f0459 -> b65fe43)

2020-09-21 Thread jihao
This is an automated email from the ASF dual-hosted git repository. jihao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 73f0459 Add Broker Reduce Time Log (#6033) add b65fe43 [TE] fix labeler config mapping and timeout

[GitHub] [incubator-pinot] adriancole opened a new pull request #6039: WIP: ServiceManager ADD_TABLE role

2020-09-21 Thread GitBox
adriancole opened a new pull request #6039: URL: https://github.com/apache/incubator-pinot/pull/6039 This is a work in progress towards #5977 Besides overall review of this sketch, this needs. [ ] how do we get a reference to the _helixResourceManager? [ ] do we need to

[GitHub] [incubator-pinot] adriancole commented on issue #5977: Allow ServiceManager to install tables prior to listening on service ports or a healthy status

2020-09-21 Thread GitBox
adriancole commented on issue #5977: URL: https://github.com/apache/incubator-pinot/issues/5977#issuecomment-696109515 ok I added a sketch for feedback. I will work on this again next week https://github.com/apache/incubator-pinot/pull/6039