[GitHub] [incubator-pinot] codecov-commenter commented on pull request #5905: Add 0.5 license and notice.

2020-08-26 Thread GitBox
codecov-commenter commented on pull request #5905: URL: https://github.com/apache/incubator-pinot/pull/5905#issuecomment-680698790 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5905?src=pr=h1) Report > Merging

[GitHub] [incubator-pinot] adriancole commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
adriancole commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-680704778 ok here is what it looks like now: ``` $ docker logs pinot 2>&1|grep 'INFO.*since launch' 07:12:45,859 INFO [main] command.StartServiceManagerCommand

[GitHub] [incubator-pinot] chenboat commented on pull request #5905: Add 0.5 license and notice.

2020-08-26 Thread GitBox
chenboat commented on pull request #5905: URL: https://github.com/apache/incubator-pinot/pull/5905#issuecomment-680687103 Thanks for the reviews. (1) org.apache.helix:helix-core:0.9.7 is still in the LICENSE-binary file. (2) cddl 1.0 should _not_ go away. I think the current runbook

[GitHub] [incubator-pinot] adriancole commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
adriancole commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-680776682 one problem is the logging of `org.apache.helix.manager.zk.ZKHelixManager`. Trying the cluster clutters the log like this even when we catch exception: ```

[GitHub] [incubator-pinot] codecov-commenter commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
codecov-commenter commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-680784776 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5917?src=pr=h1) Report > Merging

[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
codecov-commenter edited a comment on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-680784776 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5917?src=pr=h1) Report > Merging

[GitHub] [incubator-pinot] adriancole commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
adriancole commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-680750417 I'm pretty satisfied with this now ``` $ docker logs pinot 2>&1|grep 'INFO.*since launch' 08:51:48,903 INFO [main] command.StartServiceManagerCommand

[GitHub] [incubator-pinot] adriancole commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
adriancole commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-680829587 nah chicken, egg as servicemanager starts the roles. I think this is good as it gets. This is an

[GitHub] [incubator-pinot] adriancole commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
adriancole commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-680824999 Looking at the below, I'm wondering if ServiceManager itself can be deferred to start? It takes 8s ``` docker logs pinot 2>&1|grep 'INFO.*since launch'

[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5853: Add Range Indexing support for raw values

2020-08-26 Thread GitBox
KKcorps commented on a change in pull request #5853: URL: https://github.com/apache/incubator-pinot/pull/5853#discussion_r477295356 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/index/loader/invertedindex/RangeIndexHandler.java ## @@ -140,17 +145,70 @@

[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5853: Add Range Indexing support for raw values

2020-08-26 Thread GitBox
KKcorps commented on a change in pull request #5853: URL: https://github.com/apache/incubator-pinot/pull/5853#discussion_r477295778 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/RangeIndexCreator.java ## @@ -65,7 +66,7 @@ * *

[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5853: Add Range Indexing support for raw values

2020-08-26 Thread GitBox
KKcorps commented on a change in pull request #5853: URL: https://github.com/apache/incubator-pinot/pull/5853#discussion_r477295223 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/RangeIndexCreator.java ## @@ -165,26 +166,63 @@ public

[GitHub] [incubator-pinot] mr-agrwal opened a new pull request #5927: support for local.directory.sequence.id

2020-08-26 Thread GitBox
mr-agrwal opened a new pull request #5927: URL: https://github.com/apache/incubator-pinot/pull/5927 There is support for the same using PinotSparkJobLauncher. But not using the LaunchDataIngestionJobCommand. This is an

[GitHub] [incubator-pinot] kishoreg opened a new issue #5928: Add connector for Pravega

2020-08-26 Thread GitBox
kishoreg opened a new issue #5928: URL: https://github.com/apache/incubator-pinot/issues/5928 https://www.pravega.io/ Similar to Kafka connector, it will be great to add a connector for Pravega. Some info on [pravega vs

[GitHub] [incubator-pinot] mcvsubbu commented on pull request #5922: Add max qps bucket count

2020-08-26 Thread GitBox
mcvsubbu commented on pull request #5922: URL: https://github.com/apache/incubator-pinot/pull/5922#issuecomment-680976015 > > Shouldn't we be setting the value to "Max qps within a second since the time the callback was invoked last"? > > We don't want to make the metrics system

[GitHub] [incubator-pinot] fx19880617 commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
fx19880617 commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-681059064 I think this is good for now. I will take a look how to let service manager to connect to helix and decide if it's required to start controller first or it can

[incubator-pinot] branch master updated (9e65bdf -> ba55d58)

2020-08-26 Thread xiangfu
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from 9e65bdf [TE] cube api bug fix to fetch by dataset name (#5921) add ba55d58 Updating docusaurus and

[GitHub] [incubator-pinot] fx19880617 merged pull request #5918: Updating docusaurus and adding pinot navbar logos

2020-08-26 Thread GitBox
fx19880617 merged pull request #5918: URL: https://github.com/apache/incubator-pinot/pull/5918 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [incubator-pinot] samarth-gupta-traceable opened a new issue #5929: BYTES column type deserializing STRING when using ConvertPinotSegment with JSON format option

2020-08-26 Thread GitBox
samarth-gupta-traceable opened a new issue #5929: URL: https://github.com/apache/incubator-pinot/issues/5929 When I run `pinot-admin.sh ConvertPinotSegment` with JSON option, the in resulting JSON file the columns whose data type is BYTES show up as string. Now when I try to run `

[GitHub] [incubator-pinot] KKcorps commented on a change in pull request #5853: Add Range Indexing support for raw values

2020-08-26 Thread GitBox
KKcorps commented on a change in pull request #5853: URL: https://github.com/apache/incubator-pinot/pull/5853#discussion_r477349453 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/DictionaryBasedInvertedIndexCreator.java ## @@ -0,0 +1,85 @@ +/** +

[GitHub] [incubator-pinot] kishoreg edited a comment on issue #5928: Add connector for Pravega

2020-08-26 Thread GitBox
kishoreg edited a comment on issue #5928: URL: https://github.com/apache/incubator-pinot/issues/5928#issuecomment-680990156 There is some prior work we did to support other streaming systems (this was done keeping Kinesis in mind). #5359 Extend offset support for other streams

[GitHub] [incubator-pinot] kishoreg commented on issue #5928: Add connector for Pravega

2020-08-26 Thread GitBox
kishoreg commented on issue #5928: URL: https://github.com/apache/incubator-pinot/issues/5928#issuecomment-680989516 @fpj This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [incubator-pinot] kishoreg commented on issue #5928: Add connector for Pravega

2020-08-26 Thread GitBox
kishoreg commented on issue #5928: URL: https://github.com/apache/incubator-pinot/issues/5928#issuecomment-680989410 @mcvsubbu @npawar and I had an initial discussion around this topic but our understanding of Pravega is a bit weak to propose a complete solution. Flavio, it will be great

[GitHub] [incubator-pinot] kishoreg commented on issue #5928: Add connector for Pravega

2020-08-26 Thread GitBox
kishoreg commented on issue #5928: URL: https://github.com/apache/incubator-pinot/issues/5928#issuecomment-680990156 There is some prior work we did to extend support for other streaming systems (this was done keeping Kinesis in mind). #5359 Extend offset support for other streams

[GitHub] [incubator-pinot] jackjlli commented on pull request #5922: Add max qps bucket count

2020-08-26 Thread GitBox
jackjlli commented on pull request #5922: URL: https://github.com/apache/incubator-pinot/pull/5922#issuecomment-681033126 > > > Shouldn't we be setting the value to "Max qps within a second since the time the callback was invoked last"? > > > > > > We don't want to make the

[GitHub] [incubator-pinot] fx19880617 commented on a change in pull request #5927: support for local.directory.sequence.id

2020-08-26 Thread GitBox
fx19880617 commented on a change in pull request #5927: URL: https://github.com/apache/incubator-pinot/pull/5927#discussion_r477566730 ## File path:

[GitHub] [incubator-pinot] jihaozh commented on a change in pull request #5930: [TE] ThirdEye User Configuration Validation - detection and subscription yaml

2020-08-26 Thread GitBox
jihaozh commented on a change in pull request #5930: URL: https://github.com/apache/incubator-pinot/pull/5930#discussion_r477624214 ## File path: thirdeye/thirdeye-pinot/src/main/resources/validators/detection/detection-config-schema.json ## @@ -0,0 +1,214 @@ +{ + "$schema":

[GitHub] [incubator-pinot] jihaozh commented on a change in pull request #5930: [TE] ThirdEye User Configuration Validation - detection and subscription yaml

2020-08-26 Thread GitBox
jihaozh commented on a change in pull request #5930: URL: https://github.com/apache/incubator-pinot/pull/5930#discussion_r477624953 ## File path: thirdeye/thirdeye-pinot/src/main/resources/validators/subscription/subscription-config-schema.json ## @@ -0,0 +1,110 @@ +{ +

[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5930: [TE] ThirdEye User Configuration Validation - detection and subscription yaml

2020-08-26 Thread GitBox
akshayrai commented on a change in pull request #5930: URL: https://github.com/apache/incubator-pinot/pull/5930#discussion_r477627360 ## File path: thirdeye/thirdeye-pinot/src/main/resources/validators/detection/detection-config-schema.json ## @@ -0,0 +1,214 @@ +{ +

[GitHub] [incubator-pinot] Jackie-Jiang merged pull request #5931: Fix CSV and JSON converter on BYTES column

2020-08-26 Thread GitBox
Jackie-Jiang merged pull request #5931: URL: https://github.com/apache/incubator-pinot/pull/5931 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[incubator-pinot] branch master updated (dafbef1 -> ced3a70)

2020-08-26 Thread jackie
This is an automated email from the ASF dual-hosted git repository. jackie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from dafbef1 Add 0.5 license and notice. (#5905) add ced3a70 Fix CSV and JSON converter on BYTES column

[GitHub] [incubator-pinot] Jackie-Jiang closed issue #5929: BYTES column type deserializing STRING when using ConvertPinotSegment with JSON format option

2020-08-26 Thread GitBox
Jackie-Jiang closed issue #5929: URL: https://github.com/apache/incubator-pinot/issues/5929 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-pinot] akshayrai commented on a change in pull request #5930: [TE] ThirdEye User Configuration Validation - detection and subscription yaml

2020-08-26 Thread GitBox
akshayrai commented on a change in pull request #5930: URL: https://github.com/apache/incubator-pinot/pull/5930#discussion_r477627648 ## File path: thirdeye/thirdeye-pinot/src/main/resources/validators/subscription/subscription-config-schema.json ## @@ -0,0 +1,110 @@ +{ +

[incubator-pinot] branch master updated (ced3a70 -> 1b38909)

2020-08-26 Thread akshayrai09
This is an automated email from the ASF dual-hosted git repository. akshayrai09 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ced3a70 Fix CSV and JSON converter on BYTES column (#5931) add 1b38909 [TE] ThirdEye User

[GitHub] [incubator-pinot] akshayrai merged pull request #5930: [TE] ThirdEye User Configuration Validation - detection and subscription yaml

2020-08-26 Thread GitBox
akshayrai merged pull request #5930: URL: https://github.com/apache/incubator-pinot/pull/5930 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [incubator-pinot] jackjlli commented on pull request #5873: Add Hadoop counters for detecting schema mismatch

2020-08-26 Thread GitBox
jackjlli commented on pull request #5873: URL: https://github.com/apache/incubator-pinot/pull/5873#issuecomment-681185598 > Most of these are already tracked in transformers. Invalid columns etc Plus, the transformers are applied on each of the records. We don't have to do the

[GitHub] [incubator-pinot] vincentchenjl commented on a change in pull request #5920: [TE] only merge same trend in ChildKeepingMerge

2020-08-26 Thread GitBox
vincentchenjl commented on a change in pull request #5920: URL: https://github.com/apache/incubator-pinot/pull/5920#discussion_r477835580 ## File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/wrapper/ChildKeepingMergeWrapper.java ## @@ -91,10

[GitHub] [incubator-pinot] adriancole commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
adriancole commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-681200903 ok PTAL I finished polishing the exit status work and updated Javadoc This is an automated message

[GitHub] [incubator-pinot] jihaozh commented on a change in pull request #5920: [TE] only merge same trend in ChildKeepingMerge

2020-08-26 Thread GitBox
jihaozh commented on a change in pull request #5920: URL: https://github.com/apache/incubator-pinot/pull/5920#discussion_r477664239 ## File path: thirdeye/thirdeye-pinot/src/main/java/org/apache/pinot/thirdeye/detection/wrapper/ChildKeepingMergeWrapper.java ## @@ -91,10

[GitHub] [incubator-pinot] akshayrai opened a new pull request #5930: [TE] ThirdEye User Configuration Validation - detection and subscription yaml

2020-08-26 Thread GitBox
akshayrai opened a new pull request #5930: URL: https://github.com/apache/incubator-pinot/pull/5930 ThirdEye uses YAML files to manage the configuration of alerts and subscription groups. This approach enables employees across engineering and non-engineering, to configure simple to

[GitHub] [incubator-pinot] renato2099 edited a comment on pull request #5910: 5804: Prints file name of invalid json file when parsing

2020-08-26 Thread GitBox
renato2099 edited a comment on pull request #5910: URL: https://github.com/apache/incubator-pinot/pull/5910#issuecomment-681089584 I was not sure which were the code paths that could use the json parsing, that is why I added it where it actually takes place. I moved it now to the

[GitHub] [incubator-pinot] renato2099 commented on pull request #5910: 5804: Prints file name of invalid json file when parsing

2020-08-26 Thread GitBox
renato2099 commented on pull request #5910: URL: https://github.com/apache/incubator-pinot/pull/5910#issuecomment-681089584 I was not sure which were the code paths that could use the json parsing, that is why I added it where it actually takes place. I moved it now to the

[GitHub] [incubator-pinot] fx19880617 commented on pull request #5927: support for local.directory.sequence.id

2020-08-26 Thread GitBox
fx19880617 commented on pull request #5927: URL: https://github.com/apache/incubator-pinot/pull/5927#issuecomment-681105197 Can you also update the PR description of why we need to do this, e.g. multiple input files per day and you want to refresh data on same day ? And give an example

[incubator-pinot] branch master updated (ba55d58 -> dafbef1)

2020-08-26 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. from ba55d58 Updating docusaurus and adding pinot navbar logos (#5918) add dafbef1 Add 0.5 license and

[GitHub] [incubator-pinot] chenboat merged pull request #5905: Add 0.5 license and notice.

2020-08-26 Thread GitBox
chenboat merged pull request #5905: URL: https://github.com/apache/incubator-pinot/pull/5905 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [incubator-pinot] Jackie-Jiang opened a new pull request #5931: Fix CSV and JSON converter on BYTES column

2020-08-26 Thread GitBox
Jackie-Jiang opened a new pull request #5931: URL: https://github.com/apache/incubator-pinot/pull/5931 ## Description Fix #5929 Fix the issue of `PinotSegmentToCsvConverter` and `PinotSegmentToJsonConverter` not handling `BYTES` column properly Add test to test converter for all

[GitHub] [incubator-pinot] fpj commented on issue #5928: Add connector for Pravega

2020-08-26 Thread GitBox
fpj commented on issue #5928: URL: https://github.com/apache/incubator-pinot/issues/5928#issuecomment-681121098 Thanks for starting this issue, @kishoreg. I'm wondering what would be the best way to approach the implementation of a connector. I started by looking at the spi interfaces,

[GitHub] [incubator-pinot] adriancole commented on pull request #5917: Starts Broker and Server in parallel when using ServiceManager

2020-08-26 Thread GitBox
adriancole commented on pull request #5917: URL: https://github.com/apache/incubator-pinot/pull/5917#issuecomment-681180908 Thanks @fx19880617 and @kishoreg I will make one last sweep here to make sure if a role crashes the process does. then ask to merge (and ideally cut a snapshot)

[GitHub] [incubator-pinot] apucher merged pull request #5932: [TE] Cosmetic. Remove incessant DEBUG level logging from Data Layer classes

2020-08-26 Thread GitBox
apucher merged pull request #5932: URL: https://github.com/apache/incubator-pinot/pull/5932 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[incubator-pinot] branch master updated: [TE] Remove incessant DEBUG level logging (#5932)

2020-08-26 Thread apucher
This is an automated email from the ASF dual-hosted git repository. apucher pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/master by this push: new 9cd46b1 [TE] Remove incessant DEBUG

[GitHub] [incubator-pinot] mcvsubbu commented on pull request #5922: Add max qps bucket count

2020-08-26 Thread GitBox
mcvsubbu commented on pull request #5922: URL: https://github.com/apache/incubator-pinot/pull/5922#issuecomment-681183127 Discussed offline. The "clearing" of values is logical. Implementation can be that we maintain a from/to handle on the circular buffer and count only those buckets

[GitHub] [incubator-pinot] suvodeep-pyne opened a new pull request #5932: [TE] Cosmetic. Remove incessant DEBUG level logging from Data Layer classes

2020-08-26 Thread GitBox
suvodeep-pyne opened a new pull request #5932: URL: https://github.com/apache/incubator-pinot/pull/5932 Removing a few debug level logging on a couple of classes which creates a lot of noise in the server logs. Also these pollute the build logs which now shoot up to 14 MB. No

[incubator-pinot] branch release-0.5.0-rc updated: Add 0.5 license and notice. (#5905)

2020-08-26 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch release-0.5.0-rc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/release-0.5.0-rc by this push: new f443cba Add 0.5

svn commit: r41133 - /release/incubator/pinot/KEYS

2020-08-26 Thread tingchen
Author: tingchen Date: Wed Aug 26 23:46:52 2020 New Revision: 41133 Log: Update Pinot KEYS file for C650A5210408F8F4 Modified: release/incubator/pinot/KEYS Modified: release/incubator/pinot/KEYS == ---

[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #5853: Add Range Indexing support for raw values

2020-08-26 Thread GitBox
Jackie-Jiang commented on a change in pull request #5853: URL: https://github.com/apache/incubator-pinot/pull/5853#discussion_r477694366 ## File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/inv/RangeIndexCreator.java ## @@ -45,7 +46,7 @@ /**

[incubator-pinot] branch release-0.5.0-rc updated: [maven-release-plugin] prepare for next development iteration

2020-08-26 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch release-0.5.0-rc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/release-0.5.0-rc by this push: new 4a9dc63

[incubator-pinot] branch release-0.5.0-rc updated: [maven-release-plugin] prepare release release-0.5.0-rc0

2020-08-26 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a commit to branch release-0.5.0-rc in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git The following commit(s) were added to refs/heads/release-0.5.0-rc by this push: new 852c946

[incubator-pinot] annotated tag release-0.5.0-rc0 updated (852c946 -> 6af3743)

2020-08-26 Thread tingchen
This is an automated email from the ASF dual-hosted git repository. tingchen pushed a change to annotated tag release-0.5.0-rc0 in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. *** WARNING: tag release-0.5.0-rc0 was modified! *** from 852c946 (commit) to

[GitHub] [incubator-pinot] adriancole commented on pull request #5846: Adding controller healthcheck endpoint: /health

2020-08-26 Thread GitBox
adriancole commented on pull request #5846: URL: https://github.com/apache/incubator-pinot/pull/5846#issuecomment-681321552 mentioned on slack, but can we get a composite variant of this for ServiceManager? This is an

[GitHub] [incubator-pinot] guruguha commented on pull request #5718: Feature/#5390 segment indexing reload status api

2020-08-26 Thread GitBox
guruguha commented on pull request #5718: URL: https://github.com/apache/incubator-pinot/pull/5718#issuecomment-681329302 @mcvsubbu can you please review again? This is an automated message from the Apache Git Service. To