[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #5483: Add QueryContext to replace BrokerRequest in the query engine
codecov-commenter edited a comment on pull request #5483: URL: https://github.com/apache/incubator-pinot/pull/5483#issuecomment-642283701 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5483?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@2c955ce`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0.51%`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5483?src=pr=tree) ```diff @@Coverage Diff@@ ## master#5483 +/- ## = Coverage ? 44.72% = Files ? 1122 Lines ?57564 Branches ? 8618 = Hits ?25745 Misses?29559 Partials ? 2260 ``` | Flag | Coverage Δ | | |---|---|---| | #integrationtests | `44.72% <0.51%> (?)` | | | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5483?src=pr=tree) | Coverage Δ | | |---|---|---| | [...mon/request/transform/TransformExpressionTree.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVxdWVzdC90cmFuc2Zvcm0vVHJhbnNmb3JtRXhwcmVzc2lvblRyZWUuamF2YQ==) | `77.77% <0.00%> (ø)` | | | [.../core/query/request/context/ExpressionContext.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvRXhwcmVzc2lvbkNvbnRleHQuamF2YQ==) | `0.00% <0.00%> (ø)` | | | [...inot/core/query/request/context/FilterContext.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvRmlsdGVyQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ot/core/query/request/context/FunctionContext.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvRnVuY3Rpb25Db250ZXh0LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...uery/request/context/OrderByExpressionContext.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvT3JkZXJCeUV4cHJlc3Npb25Db250ZXh0LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `0.00% <0.00%> (ø)` | | | [...e/query/request/context/predicate/EqPredicate.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvcHJlZGljYXRlL0VxUHJlZGljYXRlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...e/query/request/context/predicate/InPredicate.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvcHJlZGljYXRlL0luUHJlZGljYXRlLmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../request/context/predicate/IsNotNullPredicate.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvcHJlZGljYXRlL0lzTm90TnVsbFByZWRpY2F0ZS5qYXZh) | `0.00% <0.00%> (ø)` | | | [...ery/request/context/predicate/IsNullPredicate.java](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvcHJlZGljYXRlL0lzTnVsbFByZWRpY2F0ZS5qYXZh) | `0.00% <0.00%> (ø)` | | | ... and [1131 more](https://codecov.io/gh/apache/incubator-pinot/pull/5483/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5483?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5483?src=pr=footer). Last update
[GitHub] [incubator-pinot] fx19880617 opened a new issue #5565: Timeout exceptions is not set clearly in query response.
fx19880617 opened a new issue #5565:
URL: https://github.com/apache/incubator-pinot/issues/5565
Currently we don't see timeout exception in pinot query exceptions.
Sometimes we just see pure empty response, and sometime we see the message
below, which shows no clue of this is timeout and users need to see Pinot
server log to identify the real issue is timeout.
This also prevents client side to retry on certain exceptions.
```
{"errorCode":200,"message":"QueryExecutionError:\njava.lang.RuntimeException:
Caught exception while running CombinePlanNode.\n\tat
org.apache.pinot.core.plan.CombinePlanNode.run(CombinePlanNode.java:148)\n\tat
org.apache.pinot.core.plan.InstanceResponsePlanNode.run(InstanceResponsePlanNode.java:38)\n\tat
org.apache.pinot.core.plan.GlobalPlanImplV0.execute(GlobalPlanImplV0.java:45)\n\tat
org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:213)\n\tat
org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:152)\n\tat
org.apache.pinot.core.query.scheduler.QueryScheduler.lambda$createQueryFutureTask$0(QueryScheduler.java:136)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat
shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)\n\tat
shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)\n\tat
shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat
java.lang.Thread.run(Thread.java:748)"}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on pull request #5483: Add QueryContext to replace BrokerRequest in the query engine
Jackie-Jiang commented on pull request #5483: URL: https://github.com/apache/incubator-pinot/pull/5483#issuecomment-643885256 > Approving this since overall I am onboard with the change. My only request is to please try and make one more attempt at keeping separate names unless absolutely impossible. Even if the new name is ugly, it is fine IMHO. @siddharthteotia Appended `Context` to these classes to differentiate them from the thrift classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] guruguha commented on issue #5390: Improve the experience when adding indexes and reloading
guruguha commented on issue #5390: URL: https://github.com/apache/incubator-pinot/issues/5390#issuecomment-643821124 I have a Google doc with the description and base design. Please review: Pinot [#5390](https://docs.google.com/document/d/1E_J7PxF9WtaE6ido__u0O-Emtyu0IghQicC-1eLZeRo) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] haibow merged pull request #5560: Update links for release 0.4.0 in the website
haibow merged pull request #5560: URL: https://github.com/apache/incubator-pinot/pull/5560 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch master updated: Update links for release 0.4.0 in the website (#5560)
This is an automated email from the ASF dual-hosted git repository.
haibow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
The following commit(s) were added to refs/heads/master by this push:
new 2c955ce Update links for release 0.4.0 in the website (#5560)
2c955ce is described below
commit 2c955cedd19523a5e96535e158c92d8f60e3da85
Author: haibow
AuthorDate: Sun Jun 14 10:54:17 2020 -0700
Update links for release 0.4.0 in the website (#5560)
---
website/src/pages/download.js | 31 +++
website/src/pages/index.js| 6 +++---
2 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/website/src/pages/download.js b/website/src/pages/download.js
index 28c6386..2570edd 100644
--- a/website/src/pages/download.js
+++ b/website/src/pages/download.js
@@ -44,6 +44,37 @@ function Download() {
+https://www.apache.org/dyn/closer.lua/incubator/pinot/apache-pinot-incubating-0.4.0/apache-pinot-incubating-0.4.0-bin.tar.gz;
className="panel panel--link text--center">
+
+
+
+
+ 0.4.0
+
+ https://www.apache.org/dyn/closer.lua/incubator/pinot/apache-pinot-incubating-0.4.0/apache-pinot-incubating-0.4.0-src.tar.gz;
>
+ Official source release
+
+ https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-0.4.0/apache-pinot-incubating-0.4.0-src.tar.gz.sha512;
>
+ SHA512
+
+ https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-0.4.0/apache-pinot-incubating-0.4.0-src.tar.gz.asc;
>
+ ASC
+
+
+ https://www.apache.org/dyn/closer.lua/incubator/pinot/apache-pinot-incubating-0.4.0/apache-pinot-incubating-0.4.0-bin.tar.gz;
>
+ Official binary release
+
+ https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-0.4.0/apache-pinot-incubating-0.4.0-bin.tar.gz.sha512;
>
+ SHA512
+
+ https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-0.4.0/apache-pinot-incubating-0.4.0-bin.tar.gz.asc;
>
+ ASC
+
+
+
+
+
+
https://www.apache.org/dyn/closer.lua/incubator/pinot/apache-pinot-incubating-0.3.0/apache-pinot-incubating-0.3.0-bin.tar.gz;
className="panel panel--link text--center">
diff --git a/website/src/pages/index.js b/website/src/pages/index.js
index 39139cd..999c09d 100755
--- a/website/src/pages/index.js
+++ b/website/src/pages/index.js
@@ -230,7 +230,7 @@ function Installation() {
{
-`VERSION=0.3.0\nwget
https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz\ntar
vxf apache-pinot-incubating-*-bin.tar.gz\ncd
apache-pinot-incubating-*-bin\nbin/quick-start-batch.sh`
+`VERSION=0.4.0\nwget
https://downloads.apache.org/incubator/pinot/apache-pinot-incubating-$VERSION/apache-pinot-incubating-$VERSION-bin.tar.gz\ntar
vxf apache-pinot-incubating-*-bin.tar.gz\ncd
apache-pinot-incubating-*-bin\nbin/quick-start-batch.sh`
}
@@ -277,9 +277,9 @@ function Home() {
- https://docs.pinot.apache.org/releases/0.3.0;
className={styles.indexAnnouncement}>
+ https://docs.pinot.apache.org/releases/0.4.0;
className={styles.indexAnnouncement}>
release
- v0.3 has been released! Check the release notes
+ v0.4.0 has been released! Check the release notes
{siteConfig.title}
{siteConfig.tagline}, designed to
answer OLAP queries with low latency
-
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter commented on pull request #5564: Adding support to configure log2m value for hyperloglog
codecov-commenter commented on pull request #5564: URL: https://github.com/apache/incubator-pinot/pull/5564#issuecomment-643773892 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5564?src=pr=h1) Report > Merging [#5564](https://codecov.io/gh/apache/incubator-pinot/pull/5564?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a=desc) will **increase** coverage by `0.13%`. > The diff coverage is `71.15%`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5564?src=pr=tree) ```diff @@Coverage Diff @@ ## master#5564 +/- ## == + Coverage 66.44% 66.58% +0.13% == Files1075 1105 +30 Lines 5477357024+2251 Branches 8168 8543 +375 == + Hits3639637970+1574 - Misses 1570016252 +552 - Partials 2677 2802 +125 ``` | Flag | Coverage Δ | | |---|---|---| | #integrationtests | `45.33% <51.48%> (?)` | | | #unittests | `56.85% <62.47%> (?)` | | | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5564?src=pr=tree) | Coverage Δ | | |---|---|---| | [...quota/HelixExternalViewBasedQueryQuotaManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IZWxpeEV4dGVybmFsVmlld0Jhc2VkUXVlcnlRdW90YU1hbmFnZXIuamF2YQ==) | `67.87% <0.00%> (ø)` | | | [...org/apache/pinot/common/function/FunctionInfo.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25JbmZvLmphdmE=) | `73.33% <ø> (ø)` | | | [...java/org/apache/pinot/common/segment/ReadMode.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vc2VnbWVudC9SZWFkTW9kZS5qYXZh) | `66.66% <ø> (ø)` | | | [...org/apache/pinot/common/utils/CommonConstants.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `39.02% <0.00%> (+0.92%)` | :arrow_up: | | [...troller/helix/core/retention/RetentionManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JldGVudGlvbi9SZXRlbnRpb25NYW5hZ2VyLmphdmE=) | `76.05% <0.00%> (-3.12%)` | :arrow_down: | | [...he/pinot/controller/util/AutoAddInvertedIndex.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci91dGlsL0F1dG9BZGRJbnZlcnRlZEluZGV4LmphdmE=) | `0.00% <0.00%> (ø)` | | | [.../org/apache/pinot/core/common/BaseBlockValSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vQmFzZUJsb2NrVmFsU2V0LmphdmE=) | `3.03% <0.00%> (-1.32%)` | :arrow_down: | | [.../java/org/apache/pinot/core/common/DataSource.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YVNvdXJjZS5qYXZh) | `100.00% <ø> (ø)` | | | [...ta/aggregator/DistinctCountHLLValueAggregator.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL2FnZ3JlZ2F0b3IvRGlzdGluY3RDb3VudEhMTFZhbHVlQWdncmVnYXRvci5qYXZh) | `85.71% <ø> (ø)` | | | [...data/manager/realtime/DefaultSegmentCommitter.java](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvRGVmYXVsdFNlZ21lbnRDb21taXR0ZXIuamF2YQ==) | `80.00% <ø> (ø)` | | | ... and [437 more](https://codecov.io/gh/apache/incubator-pinot/pull/5564/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5564?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5564?src=pr=footer). Last update
[incubator-pinot] branch cluster_config_log2m updated (fec0e54 -> 3799d93)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch cluster_config_log2m in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. discard fec0e54 Adding support to configure log2m value for hyperloglog add 3799d93 Adding support to configure log2m value for hyperloglog This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (fec0e54) \ N -- N -- N refs/heads/cluster_config_log2m (3799d93) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../requesthandler/BaseBrokerRequestHandler.java | 26 -- 1 file changed, 14 insertions(+), 12 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 opened a new pull request #5564: Adding support to configure log2m value for hyperloglog
fx19880617 opened a new pull request #5564: URL: https://github.com/apache/incubator-pinot/pull/5564 ## Description Current distinctCountHLL hard coded Hyperloglo object's `log2m` value to 8, which prevents users from tuning the results accuracy vs query speed. This PR extends `distinctCountHLL` to take `log2m` value as the second argument. Adding a cluster config to allow users to customize default log2m value for query. ## Release Notes * Support customized accuracy for `distinctCountHLL`, `distinctCountHLLMV` functions by adding `log2m` value as the second parameter in the function. * Adding cluster config: `default.hyperloglog.log2m` to allow user set default `log2m` value. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] branch cluster_config_log2m created (now fec0e54)
This is an automated email from the ASF dual-hosted git repository. xiangfu pushed a change to branch cluster_config_log2m in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git. at fec0e54 Adding support to configure log2m value for hyperloglog This branch includes the following new commits: new fec0e54 Adding support to configure log2m value for hyperloglog The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[incubator-pinot] 01/01: Adding support to configure log2m value for hyperloglog
This is an automated email from the ASF dual-hosted git repository.
xiangfu pushed a commit to branch cluster_config_log2m
in repository https://gitbox.apache.org/repos/asf/incubator-pinot.git
commit fec0e54e7622770cbf0ea9ce9fa63e610963037a
Author: Xiang Fu
AuthorDate: Sun Jun 14 04:26:57 2020 -0700
Adding support to configure log2m value for hyperloglog
---
.../broker/broker/helix/HelixBrokerStarter.java| 10 +++-
.../requesthandler/BaseBrokerRequestHandler.java | 62 ++
.../apache/pinot/common/utils/CommonConstants.java | 3 ++
.../helix/core/util/HelixSetupUtils.java | 1 +
.../pinot/controller/helix/ControllerTest.java | 2 +
.../DistinctCountHLLValueAggregator.java | 1 +
.../function/AggregationFunctionFactory.java | 8 +--
.../DistinctCountHLLAggregationFunction.java | 52 +++---
.../DistinctCountHLLMVAggregationFunction.java | 27 +-
.../DistinctCountRawHLLAggregationFunction.java| 5 +-
.../DistinctCountRawHLLMVAggregationFunction.java | 5 +-
.../tests/OfflineClusterIntegrationTest.java | 28 ++
12 files changed, 163 insertions(+), 41 deletions(-)
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java
index 822b37e..5bee98b 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/broker/helix/HelixBrokerStarter.java
@@ -191,11 +191,19 @@ public class HelixBrokerStarter implements
ServiceStartable {
new
HelixConfigScopeBuilder(HelixConfigScope.ConfigScopeProperty.CLUSTER).forCluster(_clusterName).build();
Map configMap = configAccessor.get(helixConfigScope, Arrays
.asList(Helix.ENABLE_CASE_INSENSITIVE_KEY,
Helix.DEPRECATED_ENABLE_CASE_INSENSITIVE_KEY,
-Broker.CONFIG_OF_ENABLE_QUERY_LIMIT_OVERRIDE));
+Broker.CONFIG_OF_ENABLE_QUERY_LIMIT_OVERRIDE,
Helix.CONFIG_OF_DEFAULT_HYPERLOGLOG_LOG2M));
if (Boolean.parseBoolean(configMap.get(Helix.ENABLE_CASE_INSENSITIVE_KEY))
|| Boolean
.parseBoolean(configMap.get(Helix.DEPRECATED_ENABLE_CASE_INSENSITIVE_KEY))) {
_brokerConf.setProperty(Helix.ENABLE_CASE_INSENSITIVE_KEY, true);
}
+String log2mStr = configMap.get(Helix.CONFIG_OF_DEFAULT_HYPERLOGLOG_LOG2M);
+if (log2mStr != null) {
+ try {
+_brokerConf.setProperty(Helix.CONFIG_OF_DEFAULT_HYPERLOGLOG_LOG2M,
Integer.parseInt(log2mStr));
+ } catch (NumberFormatException e) {
+LOGGER.warn("Unable to set Log2M value {} for DistinctCountHLL
function. {}", log2mStr, e);
+ }
+}
if
(Boolean.parseBoolean(configMap.get(Broker.CONFIG_OF_ENABLE_QUERY_LIMIT_OVERRIDE)))
{
_brokerConf.setProperty(Broker.CONFIG_OF_ENABLE_QUERY_LIMIT_OVERRIDE,
true);
}
diff --git
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
index bda87f2..584686e 100644
---
a/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
+++
b/pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
@@ -62,6 +62,7 @@ import org.apache.pinot.common.request.FilterOperator;
import org.apache.pinot.common.request.FilterQuery;
import org.apache.pinot.common.request.FilterQueryMap;
import org.apache.pinot.common.request.Identifier;
+import org.apache.pinot.common.request.Function;
import org.apache.pinot.common.request.Literal;
import org.apache.pinot.common.request.PinotQuery;
import org.apache.pinot.common.request.Selection;
@@ -74,6 +75,7 @@ import org.apache.pinot.common.utils.CommonConstants;
import org.apache.pinot.common.utils.CommonConstants.Broker;
import org.apache.pinot.common.utils.DataSchema;
import org.apache.pinot.common.utils.helix.TableCache;
+import org.apache.pinot.common.utils.request.RequestUtils;
import
org.apache.pinot.core.query.aggregation.function.AggregationFunctionUtils;
import org.apache.pinot.core.query.reduce.BrokerReduceService;
import org.apache.pinot.core.transport.ServerInstance;
@@ -112,6 +114,7 @@ public abstract class BaseBrokerRequestHandler implements
BrokerRequestHandler {
private final boolean _enableCaseInsensitive;
private final boolean _enableQueryLimitOverride;
+ private final int _hllLog2mOverride;
private final TableCache _tableCache;
public BaseBrokerRequestHandler(Configuration config, RoutingManager
routingManager,
@@ -129,6 +132,13 @@ public abstract class BaseBrokerRequestHandler implements
BrokerRequestHandler {
} else {
_tableCache = null;
}
+if
(_config.containsKey(CommonConstants.Helix.CONFIG_OF_DEFAULT_HYPERLOGLOG_LOG2M))
{
+
[GitHub] [incubator-pinot] mr-agrwal closed pull request #5562: Fix grammatical errors on Home Page
mr-agrwal closed pull request #5562: URL: https://github.com/apache/incubator-pinot/pull/5562 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter commented on pull request #5562: Fix grammatical errors on Home Page
codecov-commenter commented on pull request #5562: URL: https://github.com/apache/incubator-pinot/pull/5562#issuecomment-643741759 # [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5562?src=pr=h1) Report > Merging [#5562](https://codecov.io/gh/apache/incubator-pinot/pull/5562?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a=desc) will **decrease** coverage by `21.10%`. > The diff coverage is `51.54%`. [](https://codecov.io/gh/apache/incubator-pinot/pull/5562?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#5562 +/- ## === - Coverage 66.44% 45.34% -21.11% === Files1075 1105 +30 Lines 5477356981 +2208 Branches 8168 8529 +361 === - Hits3639625836-10560 - Misses 1570028885+13185 + Partials 2677 2260 -417 ``` | Flag | Coverage Δ | | |---|---|---| | #integrationtests | `45.34% <51.54%> (?)` | | | [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/5562?src=pr=tree) | Coverage Δ | | |---|---|---| | [...quota/HelixExternalViewBasedQueryQuotaManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IZWxpeEV4dGVybmFsVmlld0Jhc2VkUXVlcnlRdW90YU1hbmFnZXIuamF2YQ==) | `32.12% <0.00%> (-35.76%)` | :arrow_down: | | [...org/apache/pinot/common/function/FunctionInfo.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25JbmZvLmphdmE=) | `73.33% <ø> (ø)` | | | [.../apache/pinot/common/function/StringFunctions.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vU3RyaW5nRnVuY3Rpb25zLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...java/org/apache/pinot/common/segment/ReadMode.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vc2VnbWVudC9SZWFkTW9kZS5qYXZh) | `66.66% <ø> (ø)` | | | [...org/apache/pinot/common/utils/CommonConstants.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `39.02% <0.00%> (+0.92%)` | :arrow_up: | | [...e/assignment/segment/OfflineSegmentAssignment.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL2Fzc2lnbm1lbnQvc2VnbWVudC9PZmZsaW5lU2VnbWVudEFzc2lnbm1lbnQuamF2YQ==) | `22.22% <0.00%> (-68.69%)` | :arrow_down: | | [.../assignment/segment/RealtimeSegmentAssignment.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL2Fzc2lnbm1lbnQvc2VnbWVudC9SZWFsdGltZVNlZ21lbnRBc3NpZ25tZW50LmphdmE=) | `46.08% <0.00%> (-46.96%)` | :arrow_down: | | [...ntroller/helix/core/rebalance/TableRebalancer.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYmFsYW5jZS9UYWJsZVJlYmFsYW5jZXIuamF2YQ==) | `48.30% <0.00%> (-15.85%)` | :arrow_down: | | [...troller/helix/core/retention/RetentionManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JldGVudGlvbi9SZXRlbnRpb25NYW5hZ2VyLmphdmE=) | `30.98% <0.00%> (-48.19%)` | :arrow_down: | | [...he/pinot/controller/util/AutoAddInvertedIndex.java](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci91dGlsL0F1dG9BZGRJbnZlcnRlZEluZGV4LmphdmE=) | `0.00% <0.00%> (ø)` | | | ... and [849 more](https://codecov.io/gh/apache/incubator-pinot/pull/5562/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/5562?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` >
[GitHub] [incubator-pinot] fx19880617 opened a new issue #5563: Tunable Hyperloglog object size
fx19880617 opened a new issue #5563: URL: https://github.com/apache/incubator-pinot/issues/5563 Currently we default hyperloglog object log2m to 8 (accuracy = 6.5%), which doesn't fit for the use cases requires certain accuracy. Ideally we should make `log2m` configurable through: - [ ] pinot cluster level - [ ] per table level override - [ ] per query/session level override This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
[GitHub] [incubator-pinot] mr-agrwal opened a new pull request #5562: Fix grammatical errors on Home Page
mr-agrwal opened a new pull request #5562: URL: https://github.com/apache/incubator-pinot/pull/5562 ## Description Found these errors when reading about Pinot. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) * [ ] Yes (Please label as **backward-incompat**, and complete the section below on Release Notes) Does this PR fix a zero-downtime upgrade introduced earlier? * [ ] Yes (Please label this as **backward-incompat**, and complete the section below on Release Notes) Does this PR otherwise need attention when creating release notes? Things to consider: - New configuration options - Deprecation of configurations - Signature changes to public methods/interfaces - New plugins added or old plugins removed * [ ] Yes (Please label this PR as **release-notes** and complete the section on Release Notes) ## Release Notes If you have tagged this as either backward-incompat or release-notes, you MUST add text here that you would like to see appear in release notes of the next release. If you have a series of commits adding or enabling a feature, then add this section only in final commit that marks the feature completed. Refer to earlier release notes to see examples of text This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
