[GitHub] [incubator-druid] vogievetsky commented on issue #3334: Add "pailer" type functionality to overlord console
vogievetsky commented on issue #3334: Add "pailer" type functionality to overlord console URL: https://github.com/apache/incubator-druid/issues/3334#issuecomment-495479957 I believe that https://github.com/apache/incubator-druid/pull/7703 addresses this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #7506: New Router UI: No way to enable/disable datasources. confusing 'drop data' option.
vogievetsky commented on issue #7506: New Router UI: No way to enable/disable datasources. confusing 'drop data' option. URL: https://github.com/apache/incubator-druid/issues/7506#issuecomment-495479620 I believe that with the changes in #7528 this can be considered closed ( @clintropolis )  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #7751: use unminified jquery to be more friendly for source releases, fix license stuff
vogievetsky commented on issue #7751: use unminified jquery to be more friendly for source releases, fix license stuff URL: https://github.com/apache/incubator-druid/pull/7751#issuecomment-495477094 Oh damn, should I code review all of jQuery? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #7723: Web-console: add resizable split screen layout to tasks and servers views
vogievetsky commented on issue #7723: Web-console: add resizable split screen layout to tasks and servers views URL: https://github.com/apache/incubator-druid/pull/7723#issuecomment-495469068 Thank you for all the changes. This is good to go This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #7628: 0.14.2-incubating release notes
clintropolis commented on issue #7628: 0.14.2-incubating release notes URL: https://github.com/apache/incubator-druid/issues/7628#issuecomment-495464403 The vote has passed, release artifacts are currently propagating to mirrors and maven, announcement should happen in the next 24 hours or so This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis opened a new pull request #7751: use unminified jquery to be more friendly for source releases, fix license stuff
clintropolis opened a new pull request #7751: use unminified jquery to be more friendly for source releases, fix license stuff URL: https://github.com/apache/incubator-druid/pull/7751 Based on feedback from 0.14.2-incubating IPMC vote https://lists.apache.org/thread.html/19f1b682856a3975c3a3a73f75abd83d7c094215a9417b917e1331b6@%3Cgeneral.incubator.apache.org%3E Tested that old consoles still load correctly This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
svn commit: r34202 - /release/incubator/druid/0.14.2-incubating/
Author: cwylie Date: Fri May 24 03:43:07 2019 New Revision: 34202 Log: add 0.14.2 release artifacts Added: release/incubator/druid/0.14.2-incubating/ release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz (with props) release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.asc release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.sha512 release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz (with props) release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz.asc release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz.sha512 Added: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz == Binary file - no diff available. Propchange: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz -- svn:mime-type = application/octet-stream Added: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.asc == --- release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.asc (added) +++ release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.asc Fri May 24 03:43:07 2019 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEfDSTCzGq0yCo7CKx+EeUz8fyJf8FAlzV3V4ACgkQ+EeUz8fy +Jf/mqxAAk9HbyDum6TeCt/BMu1uPuCXqRTk3kMFGaYkfjKED/FO1gprLy1rTlunJ +eNidnvw3r3fcUIqH/5qA5rXlCvILXwQMvtTAcvniS0oxYTn1Z5UuoSm9Ls6tTyY2 +UYv7bJhhuAm3RAEkx71m3BjmbgcW/+6OBab94zRYQO8T180d7Dsdtts2RMoM9VYW +uJw9ThU4uGXxlNDyv97+Zm98WX90weXubCotnI8L6tHnES4mkvWUK+iPHlbbE02t +cOcvhEOzRF6W8XqujySZCu4Or3TdHJ7dFIscgutjH6AzvBL0z9oOhu7QjyoNZMW7 +Xk+RGcH5fld0L/YXiGgLRSt+timHvadI9s/DpAPTNm9xMtBVcvtV8RBSN5xd3EwZ +6YIEHq/BNVSN7KCoEBHIJ8KtGL/hGobzlMzYBkA+35dekGFFO15aP6Pabb91yhLw +3IfMDqHzvdPPkUg9uwIpxkCeIfUFC2vbu5WqDePaItveGhG2hDc0G3dE1eJsRN09 +noQfweIKWKSkNRqPinYFr4HKfn5HZZWoGFs23mFvxtuDO45Uk/XYG1cNGAh7IGoh +FE6B3QZ9IxrYGNdLBCyMRMPpSl1V3aHseRfbSLRdgUbYHD4FI7gpKNbqVf2gfqpr +U+PDxWcufMbq/HCocVN16t8Z5YOnitoRQHd2mx1suo3dMP8glPY= +=AR8Z +-END PGP SIGNATURE- Added: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.sha512 == --- release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.sha512 (added) +++ release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-bin.tar.gz.sha512 Fri May 24 03:43:07 2019 @@ -0,0 +1 @@ +0e38c74ac89c8cfdc93d5d071e1c78ba64f53241181938605bbb7da8a80eb9a86fce95b978e150afcfdf6ccd6b5678e49333ef62de201003d7577b9d9edfeca7 \ No newline at end of file Added: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz == Binary file - no diff available. Propchange: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz -- svn:mime-type = application/octet-stream Added: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz.asc == --- release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz.asc (added) +++ release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz.asc Fri May 24 03:43:07 2019 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEEfDSTCzGq0yCo7CKx+EeUz8fyJf8FAlzV3V4ACgkQ+EeUz8fy +Jf999hAAtjTOoXeGXSeGmmtREoV00rMCKh5WnvT9lZ5CV9lR20FRgt2qhGmhjqVU +0VvrVt5+JaSh9erCKBs2l9XCz4B6H4+/00ph0grncDDXj8hmjR7Tba3/6hXlf32c +9hZ4tDAeFYndEqcgdRUBboFlbqMcbi7p3HVwyA8/kwWESc9eP7RU7udcNd3yAbus +pcxMXTpHChG1sh8EcIhz/MyLVwsEHMNQPrz85tlvUTfw4XAe1CwyYaCB680TeVmz +zoqxyXtkPNKcDRh0S6ucwtEgH2m/zQivxhj21f4EChioAyozU4kA5WnTIUXOgQ2h +x+PTPQ21l1hVMZ/8dnMqhv6Svb9PdqpltepfsqtaBeYTytR9oJMXKpwkr8JVKXSY +u9np7Tpgk/WLfXGKN1ms+pB4unT4dQKxgxBsbIsXOPP9J6RpI2MSiowiH/cUBy9S +F8NvD6lD91kv0qRiMwGSgkqt5UFMp1Dn/tdIjmTo940yI7KJ00zw8IXzdyYYuvpK +wScughuira5Z2XGk913qNr+Uxkfh9TrqzwzFE5zpIctqnsbnNsLmocZxKywhggEZ +Ansctmf51eN/pO6s8Ai1A7XdA8MYTa+zAy34l42M5wnytmlltyEEqDoipsz4aem7 +6GBIiMohTM4zTYu2og06DAVO0YFZxlq2jbl6Xm1Xo9AmiMPGL1A= +=7Ph+ +-END PGP SIGNATURE- Added: release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz.sha512 == --- release/incubator/druid/0.14.2-incubating/apache-druid-0.14.2-incubating-src.tar.gz.sha512 (added) +++
[incubator-druid] tag druid-0.14.2-incubating created (now 1053684)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to tag druid-0.14.2-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. at 1053684 (commit) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7750: Add limit to timeseries docs
fjy merged pull request #7750: Add limit to timeseries docs URL: https://github.com/apache/incubator-druid/pull/7750 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Fix TOC clustering example link (#7749)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new f2e34a7 Fix TOC clustering example link (#7749) f2e34a7 is described below commit f2e34a76bdd5badfb909d7ea1042e97ea4cb1ae8 Author: Jonathan Wei AuthorDate: Thu May 23 19:41:27 2019 -0700 Fix TOC clustering example link (#7749) --- docs/content/toc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/toc.md b/docs/content/toc.md index 57ed632..e707ad0 100644 --- a/docs/content/toc.md +++ b/docs/content/toc.md @@ -48,7 +48,7 @@ layout: toc * [Clustering](/docs/VERSION/tutorials/cluster.html) * Further examples * [Single-server deployment](/docs/VERSION/operations/single-server.html) - * [Clustered deployment](/docs/VERSION/operations/example-cluster.html) + * [Clustered deployment](/docs/VERSION/tutorials/cluster.html#fresh-deployment) ## Data Ingestion * [Ingestion overview](/docs/VERSION/ingestion/index.html) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Add limit to timeseries docs (#7750)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new eb0e1a0 Add limit to timeseries docs (#7750) eb0e1a0 is described below commit eb0e1a056c1150e5a904d0fcef57188d6f2ae12c Author: Jonathan Wei AuthorDate: Thu May 23 19:41:52 2019 -0700 Add limit to timeseries docs (#7750) --- docs/content/querying/timeseriesquery.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/content/querying/timeseriesquery.md b/docs/content/querying/timeseriesquery.md index 9feef88..39fea97 100644 --- a/docs/content/querying/timeseriesquery.md +++ b/docs/content/querying/timeseriesquery.md @@ -76,6 +76,7 @@ There are 7 main parts to a timeseries query: |filter|See [Filters](../querying/filters.html)|no| |aggregations|See [Aggregations](../querying/aggregations.html)|no| |postAggregations|See [Post Aggregations](../querying/post-aggregations.html)|no| +|limit|An integer that limits the number of results. The default is unlimited.|no| |context|Can be used to modify query behavior, including [grand totals](#grand-totals) and [zero-filling](#zero-filling). See also [Context](../querying/query-context.html) for parameters that apply to all query types.|no| To pull it all together, the above query would return 2 data points, one for each day between 2012-01-01 and 2012-01-03, from the "sample\_datasource" table. Each data point would be the (long) sum of sample\_fieldName1, the (double) sum of sample\_fieldName2 and the (double) result of sample\_fieldName1 divided by sample\_fieldName2 for the filter set. The output looks like this: - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7749: Fix TOC clustering example link
fjy merged pull request #7749: Fix TOC clustering example link URL: https://github.com/apache/incubator-druid/pull/7749 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei opened a new pull request #7750: Add limit to timeseries docs
jon-wei opened a new pull request #7750: Add limit to timeseries docs URL: https://github.com/apache/incubator-druid/pull/7750 Support for `limit` was added to timeseries in #5894, this PR adds missing docs for the parameter. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei opened a new pull request #7749: Fix TOC clustering example link
jon-wei opened a new pull request #7749: Fix TOC clustering example link URL: https://github.com/apache/incubator-druid/pull/7749 Fixes the `Clustered deployment` example link in the TOC. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7725: Web-Console: add actions icon to lookups
vogievetsky commented on a change in pull request #7725: Web-Console: add
actions icon to lookups
URL: https://github.com/apache/incubator-druid/pull/7725#discussion_r287183690
##
File path: web-console/src/views/task-view/tasks-view.tsx
##
@@ -648,6 +656,7 @@ ORDER BY "rank" DESC, "created_time" DESC`);
width: 70,
filterable: false,
Cell: row => {
+ console.log(row);
Review comment:
no log please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] bkring-branch closed pull request #7748: Fix circleci
bkring-branch closed pull request #7748: Fix circleci URL: https://github.com/apache/incubator-druid/pull/7748 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] bkring-branch opened a new pull request #7748: Fix circleci
bkring-branch opened a new pull request #7748: Fix circleci URL: https://github.com/apache/incubator-druid/pull/7748 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7477: Java 9 compatible specialized class compilation
jihoonson commented on a change in pull request #7477: Java 9 compatible
specialized class compilation
URL: https://github.com/apache/incubator-druid/pull/7477#discussion_r287179219
##
File path:
core/src/main/java/org/apache/druid/java/util/common/DefineClassUtils.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import org.apache.druid.utils.JvmUtils;
+
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.security.ProtectionDomain;
+
+/**
+ * This utility class provides a thin runtime abstraction to pick between
+ * - sun.misc.Unsafe.defineClass in Java 8,
+ * - and MethodHandles.Lookup.defineClass in Java 9 and above,
+ * while still providing compile-time support for both Java 8 and Java 9+.
+ *
+ * See also {@link ByteBufferUtils}
+ */
+public class DefineClassUtils
Review comment:
Or would you please let me know if there are any other classes copied from
Kafka except ByteBufferUtils?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #7428: Add errors and state to stream supervisor status API endpoint
jon-wei commented on a change in pull request #7428: Add errors and state to stream supervisor status API endpoint URL: https://github.com/apache/incubator-druid/pull/7428#discussion_r287177797 ## File path: docs/content/development/extensions-core/kafka-ingestion.md ## @@ -214,12 +214,38 @@ offsets as reported by Kafka, the consumer lag per partition, as well as the agg consumer lag per partition may be reported as negative values if the supervisor has not received a recent latest offset response from Kafka. The aggregate lag value will always be >= 0. +The status report also contains the supervisor's state and a list of recently thrown exceptions (whose max size can be +controlled using the `druid.supervisor.maxStoredExceptionEvents` config parameter). The list of states is as +follows: + +|State|Description| +|-|---| +|UNHEALTHY_SUPERVISOR|The supervisor has encountered errors on the past `druid.supervisor.unhealthinessThreshold` iterations| +|UNHEALTHY_TASKS|The last `druid.supervisor.taskUnhealthinessThreshold` tasks have all failed| +|UNABLE_TO_CONNECT_TO_STREAM|The supervisor is encountering connectivity issues with Kafka and has not successfully connected in the past| +|LOST_CONTACT_WITH_STREAM|The supervisor is encountering connectivity issues with Kafka but has successfully connected in the past| +|PENDING (first iteration only)|The supervisor has been initialized and hasn't started connecting to the stream| +|CONNECTING_TO_STREAM (first iteration only)|The supervisor is trying to connect to the stream and update partition data| +|DISCOVERING_INITIAL_TASKS (first iteration only)|The supervisor is discovering already-running tasks| +|CREATING_TASKS (first iteration only)|The supervisor is creating tasks and discovering state| +|RUNNING|The supervisor has started tasks and is waiting for taskDuration to elapse| +|SUSPENDED|The supervisor has been suspended| +|STOPPING|The supervisor is stopping| + +States marked with "first iteration only" only occur on the supervisor's first iteration at startup or after suspension. Review comment: Suggest adding a short high-level summary of the Kafka/Kinesis supervisor's `runInternal()` loop. The info is kind of there implicitly in the ordering of the states above, but I think a more explicit description of the per-iteration sequence would be useful This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #7428: Add errors and state to stream supervisor status API endpoint
jon-wei commented on a change in pull request #7428: Add errors and state to stream supervisor status API endpoint URL: https://github.com/apache/incubator-druid/pull/7428#discussion_r287175986 ## File path: docs/content/development/extensions-core/kafka-ingestion.md ## @@ -214,12 +214,38 @@ offsets as reported by Kafka, the consumer lag per partition, as well as the agg consumer lag per partition may be reported as negative values if the supervisor has not received a recent latest offset response from Kafka. The aggregate lag value will always be >= 0. +The status report also contains the supervisor's state and a list of recently thrown exceptions (whose max size can be +controlled using the `druid.supervisor.maxStoredExceptionEvents` config parameter). The list of states is as +follows: + +|State|Description| +|-|---| +|UNHEALTHY_SUPERVISOR|The supervisor has encountered errors on the past `druid.supervisor.unhealthinessThreshold` iterations| Review comment: Suggest adding some info about the difference between the basic states and the detailed implementation-specific states, and how the detailed implementation states here map to basic states This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7477: Java 9 compatible specialized class compilation
jihoonson commented on a change in pull request #7477: Java 9 compatible
specialized class compilation
URL: https://github.com/apache/incubator-druid/pull/7477#discussion_r287178548
##
File path:
core/src/main/java/org/apache/druid/java/util/common/DefineClassUtils.java
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import org.apache.druid.utils.JvmUtils;
+
+import java.lang.invoke.MethodHandle;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.security.ProtectionDomain;
+
+/**
+ * This utility class provides a thin runtime abstraction to pick between
+ * - sun.misc.Unsafe.defineClass in Java 8,
+ * - and MethodHandles.Lookup.defineClass in Java 9 and above,
+ * while still providing compile-time support for both Java 8 and Java 9+.
+ *
+ * See also {@link ByteBufferUtils}
+ */
+public class DefineClassUtils
Review comment:
Hi @xvrl, I'm trying to sort out what are missing in LICENSE and NOTICE
files. Are this class and `UnsafeUtils` copied from Kafka?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7730: add support for multi-value string dimensions for 'HllSketch' build aggregator
fjy merged pull request #7730: add support for multi-value string dimensions for 'HllSketch' build aggregator URL: https://github.com/apache/incubator-druid/pull/7730 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: add support for multi-value string dimensions for HllSketch build aggregator (#7730)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new eef6961 add support for multi-value string dimensions for HllSketch
build aggregator (#7730)
eef6961 is described below
commit eef69619d39f26094963fc68ab3d51cb84bd66ee
Author: Clint Wylie
AuthorDate: Thu May 23 17:07:32 2019 -0700
add support for multi-value string dimensions for HllSketch build
aggregator (#7730)
---
.../datasketches/hll/HllSketchBuildAggregator.java | 8 +
.../datasketches/hll/GenerateTestData.java | 16 +
.../datasketches/hll/HllSketchAggregatorTest.java | 53 ++-
.../src/test/resources/hll/hll_raw.tsv | 400 ++---
.../src/test/resources/hll/hll_sketches.tsv| 200 +--
5 files changed, 372 insertions(+), 305 deletions(-)
diff --git
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildAggregator.java
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildAggregator.java
index 66c2036..4f77260 100644
---
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildAggregator.java
+++
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildAggregator.java
@@ -25,6 +25,8 @@ import org.apache.druid.java.util.common.IAE;
import org.apache.druid.query.aggregation.Aggregator;
import org.apache.druid.segment.ColumnValueSelector;
+import java.util.List;
+
/**
* This aggregator builds sketches from raw data.
* The input column can contain identifiers of type string, char[], byte[] or
any numeric type.
@@ -99,6 +101,12 @@ public class HllSketchBuildAggregator implements Aggregator
sketch.update(((Number) value).doubleValue());
} else if (value instanceof String) {
sketch.update(((String) value).toCharArray());
+} else if (value instanceof List) {
+ // noinspection unchecked
+ List list = (List) value;
+ for (String v : list) {
+sketch.update(v.toCharArray());
+ }
} else if (value instanceof char[]) {
sketch.update((char[]) value);
} else if (value instanceof byte[]) {
diff --git
a/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/GenerateTestData.java
b/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/GenerateTestData.java
index f877cef..3a40aee 100644
---
a/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/GenerateTestData.java
+++
b/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/GenerateTestData.java
@@ -70,6 +70,14 @@ class GenerateTestData
out.write("\t");
out.write(dimension);
out.write("\t");
+int parsed = Integer.parseInt(dimension);
+for (int i = parsed; i < parsed + 5; i++) {
+ out.write(Integer.toString(i));
+ if (i + 1 < parsed + 5) {
+out.write(",");
+ }
+}
+out.write("\t");
out.write(Integer.toString(id));
out.newLine();
}
@@ -80,6 +88,14 @@ class GenerateTestData
out.write("\t");
out.write(dimension);
out.write("\t");
+int parsed = Integer.parseInt(dimension);
+for (int i = parsed; i < parsed + 5; i++) {
+ out.write(Integer.toString(i));
+ if (i + 1 < parsed + 5) {
+out.write(",");
+ }
+}
+out.write("\t");
out.write(StringUtils.encodeBase64String(sketch.toCompactByteArray()));
out.newLine();
}
diff --git
a/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchAggregatorTest.java
b/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchAggregatorTest.java
index f0b995e..0490b34 100644
---
a/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchAggregatorTest.java
+++
b/extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchAggregatorTest.java
@@ -74,11 +74,12 @@ public class HllSketchAggregatorTest
"\"format\": \"tsv\",",
"\"timestampSpec\": {\"column\": \"timestamp\", \"format\":
\"MMdd\"},",
"\"dimensionsSpec\": {",
-" \"dimensions\": [\"dim\"],",
+" \"dimensions\": [\"dim\", \"multiDim\"],",
" \"dimensionExclusions\": [],",
" \"spatialDimensions\": []",
"},",
-"\"columns\": [\"timestamp\", \"dim\", \"sketch\"]",
+"\"columns\": [\"timestamp\", \"dim\", \"multiDim\",
[incubator-druid] branch 0.15.0-incubating updated: allow quantiles merge aggregator to also accept doubles (#7718) (#7743)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch 0.15.0-incubating
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.15.0-incubating by this push:
new ea3de8c allow quantiles merge aggregator to also accept doubles
(#7718) (#7743)
ea3de8c is described below
commit ea3de8cac5c82173d0563416679fef2e3e7beeb2
Author: Clint Wylie
AuthorDate: Thu May 23 17:04:08 2019 -0700
allow quantiles merge aggregator to also accept doubles (#7718) (#7743)
* allow quantiles merge aggregator to also accept doubles
* consolidate dupe
* import
---
.../quantiles/DoublesSketchBuildAggregator.java| 3 -
.../quantiles/DoublesSketchMergeAggregator.java| 23 +++--
.../DoublesSketchMergeBufferAggregator.java| 11 +--
.../sql/DoublesSketchSqlAggregatorTest.java| 106 +
4 files changed, 106 insertions(+), 37 deletions(-)
diff --git
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
index bd46fc5..18f94a9 100644
---
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
+++
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
@@ -28,14 +28,12 @@ public class DoublesSketchBuildAggregator implements
Aggregator
{
private final ColumnValueSelector valueSelector;
- private final int size;
private UpdateDoublesSketch sketch;
public DoublesSketchBuildAggregator(final ColumnValueSelector
valueSelector, final int size)
{
this.valueSelector = valueSelector;
-this.size = size;
sketch = DoublesSketch.builder().setK(size).build();
}
@@ -68,5 +66,4 @@ public class DoublesSketchBuildAggregator implements
Aggregator
{
sketch = null;
}
-
}
diff --git
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
index 325a6f2..4598048 100644
---
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
+++
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
@@ -27,10 +27,10 @@ import org.apache.druid.segment.ColumnValueSelector;
public class DoublesSketchMergeAggregator implements Aggregator
{
- private final ColumnValueSelector selector;
+ private final ColumnValueSelector selector;
private DoublesUnion union;
- public DoublesSketchMergeAggregator(final ColumnValueSelector
selector, final int k)
+ public DoublesSketchMergeAggregator(final ColumnValueSelector selector,
final int k)
{
this.selector = selector;
union = DoublesUnion.builder().setMaxK(k).build();
@@ -39,13 +39,10 @@ public class DoublesSketchMergeAggregator implements
Aggregator
@Override
public synchronized void aggregate()
{
-final DoublesSketch sketch = selector.getObject();
-if (sketch == null) {
- return;
-}
-union.update(sketch);
+updateUnion(selector, union);
}
+
@Override
public synchronized Object get()
{
@@ -70,4 +67,16 @@ public class DoublesSketchMergeAggregator implements
Aggregator
union = null;
}
+ static void updateUnion(ColumnValueSelector selector, DoublesUnion union)
+ {
+final Object object = selector.getObject();
+if (object == null) {
+ return;
+}
+if (object instanceof DoublesSketch) {
+ union.update((DoublesSketch) object);
+} else {
+ union.update(selector.getDouble());
+}
+ }
}
diff --git
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
index ffe9009..f5a1e9d 100644
---
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
+++
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
@@ -20,7 +20,6 @@
package org.apache.druid.query.aggregation.datasketches.quantiles;
import com.yahoo.memory.WritableMemory;
-import com.yahoo.sketches.quantiles.DoublesSketch;
import
[GitHub] [incubator-druid] fjy merged pull request #7743: [Backport] allow quantiles merge aggregator to also accept doubles
fjy merged pull request #7743: [Backport] allow quantiles merge aggregator to also accept doubles URL: https://github.com/apache/incubator-druid/pull/7743 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch 0.15.0-incubating updated: fix AggregatorFactory.finalizeComputation implementations to be ok with null inputs (#7731) (#7744)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch 0.15.0-incubating
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.15.0-incubating by this push:
new 8f4a690 fix AggregatorFactory.finalizeComputation implementations to
be ok with null inputs (#7731) (#7744)
8f4a690 is described below
commit 8f4a69024ebb900a313226dbe0c0d78ebd13abef
Author: Clint Wylie
AuthorDate: Thu May 23 17:03:56 2019 -0700
fix AggregatorFactory.finalizeComputation implementations to be ok with
null inputs (#7731) (#7744)
* AggregatorFactory finalizeComputation is nullable with nullable input,
make implementations honor this
* fixes
---
.../aggregation/distinctcount/DistinctCountAggregatorFactory.java | 4 +++-
.../momentsketch/aggregator/MomentSketchAggregatorFactory.java | 3 ++-
.../apache/druid/query/movingaverage/AveragerFactoryWrapper.java | 4 +++-
.../apache/druid/query/aggregation/TimestampAggregatorFactory.java | 6 --
.../aggregation/datasketches/hll/HllSketchAggregatorFactory.java | 6 +-
.../datasketches/quantiles/DoublesSketchAggregatorFactory.java | 5 +++--
.../datasketches/theta/SketchMergeAggregatorFactory.java | 7 ++-
.../datasketches/tuple/ArrayOfDoublesSketchAggregatorFactory.java | 5 +++--
.../query/aggregation/bloom/BloomFilterAggregatorFactory.java | 3 ++-
.../histogram/ApproximateHistogramAggregatorFactory.java | 5 +++--
.../query/aggregation/variance/VarianceAggregatorFactory.java | 6 --
.../org/apache/druid/query/aggregation/CountAggregatorFactory.java | 4 +++-
.../apache/druid/query/aggregation/FilteredAggregatorFactory.java | 4 +++-
.../apache/druid/query/aggregation/HistogramAggregatorFactory.java | 5 +++--
.../druid/query/aggregation/JavaScriptAggregatorFactory.java | 4 +++-
.../druid/query/aggregation/SuppressedAggregatorFactory.java | 3 ++-
.../aggregation/cardinality/CardinalityAggregatorFactory.java | 5 +++--
.../query/aggregation/first/StringFirstAggregatorFactory.java | 6 --
.../aggregation/hyperloglog/HyperUniquesAggregatorFactory.java | 6 --
.../druid/query/aggregation/last/StringLastAggregatorFactory.java | 6 --
20 files changed, 67 insertions(+), 30 deletions(-)
diff --git
a/extensions-contrib/distinctcount/src/main/java/org/apache/druid/query/aggregation/distinctcount/DistinctCountAggregatorFactory.java
b/extensions-contrib/distinctcount/src/main/java/org/apache/druid/query/aggregation/distinctcount/DistinctCountAggregatorFactory.java
index 524cc37..08f5136 100644
---
a/extensions-contrib/distinctcount/src/main/java/org/apache/druid/query/aggregation/distinctcount/DistinctCountAggregatorFactory.java
+++
b/extensions-contrib/distinctcount/src/main/java/org/apache/druid/query/aggregation/distinctcount/DistinctCountAggregatorFactory.java
@@ -35,6 +35,7 @@ import org.apache.druid.query.dimension.DefaultDimensionSpec;
import org.apache.druid.segment.ColumnSelectorFactory;
import org.apache.druid.segment.DimensionSelector;
+import javax.annotation.Nullable;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.Comparator;
@@ -147,8 +148,9 @@ public class DistinctCountAggregatorFactory extends
AggregatorFactory
return object;
}
+ @Nullable
@Override
- public Object finalizeComputation(Object object)
+ public Object finalizeComputation(@Nullable Object object)
{
return object;
}
diff --git
a/extensions-contrib/momentsketch/src/main/java/org/apache/druid/query/aggregation/momentsketch/aggregator/MomentSketchAggregatorFactory.java
b/extensions-contrib/momentsketch/src/main/java/org/apache/druid/query/aggregation/momentsketch/aggregator/MomentSketchAggregatorFactory.java
index 918ad3e..1abe6a1 100644
---
a/extensions-contrib/momentsketch/src/main/java/org/apache/druid/query/aggregation/momentsketch/aggregator/MomentSketchAggregatorFactory.java
+++
b/extensions-contrib/momentsketch/src/main/java/org/apache/druid/query/aggregation/momentsketch/aggregator/MomentSketchAggregatorFactory.java
@@ -206,8 +206,9 @@ public class MomentSketchAggregatorFactory extends
AggregatorFactory
);
}
+ @Nullable
@Override
- public Object finalizeComputation(Object object)
+ public Object finalizeComputation(@Nullable Object object)
{
return object;
}
diff --git
a/extensions-contrib/moving-average-query/src/main/java/org/apache/druid/query/movingaverage/AveragerFactoryWrapper.java
b/extensions-contrib/moving-average-query/src/main/java/org/apache/druid/query/movingaverage/AveragerFactoryWrapper.java
index f6f1d90..9e4118c 100644
---
a/extensions-contrib/moving-average-query/src/main/java/org/apache/druid/query/movingaverage/AveragerFactoryWrapper.java
+++
[GitHub] [incubator-druid] fjy merged pull request #7744: [Backport] fix AggregatorFactory.finalizeComputation implementations to be ok with null inputs
fjy merged pull request #7744: [Backport] fix AggregatorFactory.finalizeComputation implementations to be ok with null inputs URL: https://github.com/apache/incubator-druid/pull/7744 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #6214: Add [disable] button to overlord console for remote workers
clintropolis commented on issue #6214: Add [disable] button to overlord console for remote workers URL: https://github.com/apache/incubator-druid/issues/6214#issuecomment-495423914 Resolved by #7642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis closed issue #6214: Add [disable] button to overlord console for remote workers
clintropolis closed issue #6214: Add [disable] button to overlord console for remote workers URL: https://github.com/apache/incubator-druid/issues/6214 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Web console - add enable/disable actions for middle manager workers (#7642)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new cbdac49 Web console - add enable/disable actions for middle manager
workers (#7642)
cbdac49 is described below
commit cbdac49ab393c68f8f17d6cdcc604897d8843f37
Author: Bartosz Ługowski
AuthorDate: Fri May 24 01:47:23 2019 +0200
Web console - add enable/disable actions for middle manager workers (#7642)
* Overlord console - add enable/disable button for remote workers.
* Overlord console - add proxy for remote workers API.
* WorkerResourceTest - revert newline change.
* Remote worker proxy tests - remove empty line.
* Refactor remote worker proxy for readability and security
* Rename method in remote task runner tests for readability
* Remove enable/disable button for remote workers from old web console
* Add enable/disable actions for middle manager worker in new web console
* Fix variable type
* Add worker task runner query adapter
* Fix web console tests: segments-view, servers-view
* Fix overlord resource tests
---
.../druid/indexing/overlord/WorkerTaskRunner.java | 6 +
.../overlord/WorkerTaskRunnerQueryAdapter.java | 133 +++
.../indexing/overlord/http/OverlordResource.java | 48 ++-
.../overlord/WorkerTaskRunnerQueryAdpaterTest.java | 197 ++
.../overlord/http/OverlordResourceTest.java| 403 +
.../druid/indexing/overlord/http/OverlordTest.java | 5 +-
web-console/README.md | 1 +
.../__snapshots__/segments-view.spec.tsx.snap | 317
.../src/views/segments-view/segments-view.spec.tsx | 8 +-
.../__snapshots__/servers-view.spec.tsx.snap | 41 +++
.../src/views/servers-view/servers-view.tsx| 238
11 files changed, 1018 insertions(+), 379 deletions(-)
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/WorkerTaskRunner.java
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/WorkerTaskRunner.java
index c0620f8..6520de2 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/WorkerTaskRunner.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/WorkerTaskRunner.java
@@ -30,6 +30,12 @@ import java.util.Collection;
@PublicApi
public interface WorkerTaskRunner extends TaskRunner
{
+ enum ActionType
+ {
+ENABLE,
+DISABLE
+ }
+
/**
* List of known workers who can accept tasks for running
*/
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/WorkerTaskRunnerQueryAdapter.java
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/WorkerTaskRunnerQueryAdapter.java
new file mode 100644
index 000..7656761
--- /dev/null
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/WorkerTaskRunnerQueryAdapter.java
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord;
+
+import com.google.common.base.Optional;
+import com.google.common.base.Throwables;
+import com.google.common.collect.Iterables;
+import io.netty.handler.timeout.TimeoutException;
+import org.apache.druid.guice.annotations.EscalatedGlobal;
+import org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.HttpClient;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.java.util.http.client.response.StatusResponseHandler;
+import org.apache.druid.java.util.http.client.response.StatusResponseHolder;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.jboss.netty.handler.codec.http.HttpResponseStatus;
+
+import javax.inject.Inject;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+import java.util.concurrent.ExecutionException;
+
+public class
[GitHub] [incubator-druid] clintropolis merged pull request #7642: Web console - add enable/disable actions for middle manager workers
clintropolis merged pull request #7642: Web console - add enable/disable actions for middle manager workers URL: https://github.com/apache/incubator-druid/pull/7642 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7725: Web-Console: add actions icon to lookups
vogievetsky commented on a change in pull request #7725: Web-Console: add
actions icon to lookups
URL: https://github.com/apache/incubator-druid/pull/7725#discussion_r287172048
##
File path: web-console/src/views/lookups-view/lookups-view.tsx
##
@@ -54,9 +56,11 @@ export interface LookupsViewState {
lookupEditSpec: string;
isEdit: boolean;
allLookupTiers: string[];
+ lookupActionDialogId: string | null;
Review comment:
clean this up, not needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7725: Web-Console: add actions icon to lookups
vogievetsky commented on a change in pull request #7725: Web-Console: add
actions icon to lookups
URL: https://github.com/apache/incubator-druid/pull/7725#discussion_r287171719
##
File path: web-console/src/dialogs/table-action-dialog/table-action-dialog.scss
##
@@ -36,6 +36,9 @@ $side-bar-width: 120px;
}
}
+ .bp3-button{
+text-align: center;
Review comment:
what is this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287170242
##
File path: web-console/src/views/task-view/tasks-view.tsx
##
@@ -279,9 +279,17 @@ ORDER BY "rank" DESC, "created_time" DESC`);
this.taskQueryManager.rerunLastQuery();
}
- private getSupervisorActions(id: string, supervisorSuspended: boolean):
BasicAction[] {
-return [
- {
+ private getSupervisorActions(id: string, supervisorSuspended: boolean, type:
string): BasicAction[] {
+const actions = [];
+if (type === 'kafka' || type === 'kinesis') {
+ actions.push(
+{
+ icon: IconNames.WRENCH,
Review comment:
Please use the same icon as the data loader
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287170344
##
File path: web-console/src/views/task-view/tasks-view.tsx
##
@@ -503,18 +513,23 @@ ORDER BY "rank" DESC, "created_time" DESC`);
;
}
- // --
-
- private getTaskActions(id: string, status: string): BasicAction[] {
-if (status !== 'RUNNING' && status !== 'WAITING' && status !== 'PENDING')
return [];
-return [
- {
-icon: IconNames.CROSS,
+ private getTaskActions(id: string, status: string, type: string):
BasicAction[] {
+const actions = [];
+if (type === 'index' || type === 'index_parallel') {
+ actions.push({
+title: 'Open in data loader',
Review comment:
Icon please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287170132
##
File path: web-console/src/views/load-data-view/load-data-view.tsx
##
@@ -2348,11 +2349,27 @@ export class LoadDataView extends
React.Component => {
+try {
+ const resp = await
axios.get(`/druid/indexer/v1/supervisor/${this.props.initSupervisorId}`);
Review comment:
Please add a loading state
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287170096
##
File path: web-console/src/views/load-data-view/load-data-view.tsx
##
@@ -2348,11 +2349,27 @@ export class LoadDataView extends
React.Component => {
+try {
+ const resp = await
axios.get(`/druid/indexer/v1/supervisor/${this.props.initSupervisorId}`);
+ this.updateSpec(resp.data);
+ this.updateStage('json-spec');
+} catch (e) {
+}
+ }
+
+ private getTaskJson = async (): Promise => {
+try {
+ const resp = await
axios.get(`/druid/indexer/v1/task/${this.props.initTaskId}`);
+ this.updateSpec(resp.data.payload.spec);
+ this.updateStage('json-spec');
+} catch (e) {
Review comment:
Please add an error state
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] rae89 closed issue #7467: Historical node issue with segments being loaded when running locally.
rae89 closed issue #7467: Historical node issue with segments being loaded when running locally. URL: https://github.com/apache/incubator-druid/issues/7467 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch clintropolis-patch-1 deleted (was 3100c61)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch clintropolis-patch-1 in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. was 3100c61 [Backport] OrcStructConvert fix The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch 0.15.0-incubating updated: [Backport] OrcStructConvert fix (#7742)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch 0.15.0-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/0.15.0-incubating by this push: new 2441394 [Backport] OrcStructConvert fix (#7742) 2441394 is described below commit 2441394fce2d430eda16cccdc4d9f653475c2d54 Author: Clint Wylie AuthorDate: Thu May 23 16:36:53 2019 -0700 [Backport] OrcStructConvert fix (#7742) from #7738, this is a performance issue --- .../main/java/org/apache/druid/data/input/orc/OrcStructConverter.java | 2 -- 1 file changed, 2 deletions(-) diff --git a/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java b/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java index 20fbf06..69d113d 100644 --- a/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java +++ b/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java @@ -166,8 +166,6 @@ public class OrcStructConverter fieldIndexCache.put(fields.get(i), i); } } -WritableComparable wc = struct.getFieldValue(fieldName); - int fieldIndex = fieldIndexCache.getOrDefault(fieldName, -1); return convertField(struct, fieldIndex); - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7742: [Backport] OrcStructConverter fix
fjy merged pull request #7742: [Backport] OrcStructConverter fix URL: https://github.com/apache/incubator-druid/pull/7742 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7735: Remove LegacyKafkaIndexTaskRunner
jihoonson commented on issue #7735: Remove LegacyKafkaIndexTaskRunner URL: https://github.com/apache/incubator-druid/pull/7735#issuecomment-495419811 @gianm thanks for reminding me. I think I have to do some tests before adding the draft for release notes. I'm especially not sure about what happens if a task which was running with an old version is restored as a new version. I think it should just work as expected with the new version, but will try to make sure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7723: Web-console: add resizable split screen layout to tasks and servers views
vogievetsky commented on a change in pull request #7723: Web-console: add
resizable split screen layout to tasks and servers views
URL: https://github.com/apache/incubator-druid/pull/7723#discussion_r287163630
##
File path: web-console/src/views/servers-view/servers-view.tsx
##
@@ -21,18 +21,20 @@ import { IconNames } from '@blueprintjs/icons';
import axios from 'axios';
import { sum } from 'd3-array';
import * as React from 'react';
+import SplitterLayout from 'react-splitter-layout';
import ReactTable from 'react-table';
import { Filter } from 'react-table';
import { TableColumnSelection, ViewControlBar } from '../../components/index';
import {
addFilter,
formatBytes,
- formatBytesCompact, LocalStorageKeys,
+ formatBytesCompact, localStorageGet, LocalStorageKeys, localStorageSet,
queryDruidSql,
QueryManager, TableColumnSelectionHandler
} from '../../utils';
+import 'react-splitter-layout/lib/index.css';
Review comment:
import this only once in entry.scss
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7723: Web-console: add resizable split screen layout to tasks and servers views
vogievetsky commented on a change in pull request #7723: Web-console: add
resizable split screen layout to tasks and servers views
URL: https://github.com/apache/incubator-druid/pull/7723#discussion_r287164070
##
File path: web-console/src/views/task-view/tasks-view.tsx
##
@@ -697,59 +705,71 @@ ORDER BY "rank" DESC, "created_time" DESC`);
/>
;
-return
-
- this.supervisorQueryManager.rerunLastQuery()}
-/>
- this.setState({ supervisorSpecDialogOpen: true })}
-/>
-
supervisorTableColumnSelectionHandler.changeTableColumnSelection(column)}
-
tableColumnsHidden={supervisorTableColumnSelectionHandler.hiddenColumns}
-/>
-
- {this.renderSupervisorTable()}
-
-
-
-
-Group by
-
-
this.setState({ groupTasksBy: null })}>None
-
this.setState({ groupTasksBy: 'type' })}>Type
-
this.setState({ groupTasksBy: 'datasource' })}>Datasource
-
this.setState({ groupTasksBy: 'status' })}>Status
-
- this.taskQueryManager.rerunLastQuery()}
-/>
-{
- !noSqlMode &&
- goToSql(this.taskQueryManager.getLastQuery())}
- />
-}
-
-
-
-
taskTableColumnSelectionHandler.changeTableColumnSelection(column)}
- tableColumnsHidden={taskTableColumnSelectionHandler.hiddenColumns}
-/>
-
- {this.renderTaskTable()}
+// tslint:disable-next-line:radix
+return<>
Review comment:
add space please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7723: Web-console: add resizable split screen layout to tasks and servers views
vogievetsky commented on a change in pull request #7723: Web-console: add
resizable split screen layout to tasks and servers views
URL: https://github.com/apache/incubator-druid/pull/7723#discussion_r287164040
##
File path: web-console/src/views/task-view/tasks-view.tsx
##
@@ -697,59 +705,71 @@ ORDER BY "rank" DESC, "created_time" DESC`);
/>
;
-return
-
- this.supervisorQueryManager.rerunLastQuery()}
-/>
- this.setState({ supervisorSpecDialogOpen: true })}
-/>
-
supervisorTableColumnSelectionHandler.changeTableColumnSelection(column)}
-
tableColumnsHidden={supervisorTableColumnSelectionHandler.hiddenColumns}
-/>
-
- {this.renderSupervisorTable()}
-
-
-
-
-Group by
-
-
this.setState({ groupTasksBy: null })}>None
-
this.setState({ groupTasksBy: 'type' })}>Type
-
this.setState({ groupTasksBy: 'datasource' })}>Datasource
-
this.setState({ groupTasksBy: 'status' })}>Status
-
- this.taskQueryManager.rerunLastQuery()}
-/>
-{
- !noSqlMode &&
- goToSql(this.taskQueryManager.getLastQuery())}
- />
-}
-
-
-
-
taskTableColumnSelectionHandler.changeTableColumnSelection(column)}
- tableColumnsHidden={taskTableColumnSelectionHandler.hiddenColumns}
-/>
-
- {this.renderTaskTable()}
+// tslint:disable-next-line:radix
Review comment:
this is not needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7723: Web-console: add resizable split screen layout to tasks and servers views
vogievetsky commented on a change in pull request #7723: Web-console: add
resizable split screen layout to tasks and servers views
URL: https://github.com/apache/incubator-druid/pull/7723#discussion_r287161036
##
File path: web-console/src/views/servers-view/servers-view.tsx
##
@@ -415,49 +424,58 @@ WHERE "server_type" = 'historical'`);
const { groupByTier } = this.state;
const { serverTableColumnSelectionHandler,
middleManagerTableColumnSelectionHandler } = this;
-return
-
- this.serverQueryManager.rerunLastQuery()}
-/>
-{
- !noSqlMode &&
+return
+
+
goToSql(this.serverQueryManager.getLastQuery())}
+icon={IconNames.REFRESH}
+text="Refresh"
+onClick={() => this.serverQueryManager.rerunLastQuery()}
/>
-}
- this.setState({ groupByTier: !groupByTier })}
-/>
-
serverTableColumnSelectionHandler.changeTableColumnSelection(column)}
- tableColumnsHidden={serverTableColumnSelectionHandler.hiddenColumns}
-/>
-
- {this.renderServersTable()}
-
-
-
-
- this.middleManagerQueryManager.rerunLastQuery()}
-/>
-
middleManagerTableColumnSelectionHandler.changeTableColumnSelection(column)}
-
tableColumnsHidden={middleManagerTableColumnSelectionHandler.hiddenColumns}
-/>
-
- {this.renderMiddleManagerTable()}
-;
+ {
+!noSqlMode &&
+ goToSql(this.serverQueryManager.getLastQuery())}
+/>
+ }
+ this.setState({ groupByTier: !groupByTier })}
+ />
+
serverTableColumnSelectionHandler.changeTableColumnSelection(column)}
+
tableColumnsHidden={serverTableColumnSelectionHandler.hiddenColumns}
+ />
+
+{this.renderServersTable()}
+
+
Review comment:
`bottom-pane` plz
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Remove obsolete isExcluded config from Kerberos authenticator (#7745)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new ec4d09a Remove obsolete isExcluded config from Kerberos authenticator
(#7745)
ec4d09a is described below
commit ec4d09a02f81d312f677b0aa7c439a23891addef
Author: Jonathan Wei
AuthorDate: Thu May 23 16:00:05 2019 -0700
Remove obsolete isExcluded config from Kerberos authenticator (#7745)
---
.../development/extensions-core/druid-kerberos.md | 5 +++-
.../security/kerberos/KerberosAuthenticator.java | 30 ++
2 files changed, 6 insertions(+), 29 deletions(-)
diff --git a/docs/content/development/extensions-core/druid-kerberos.md
b/docs/content/development/extensions-core/druid-kerberos.md
index 46af7f4..99d6e45 100644
--- a/docs/content/development/extensions-core/druid-kerberos.md
+++ b/docs/content/development/extensions-core/druid-kerberos.md
@@ -54,13 +54,16 @@ The configuration examples in the rest of this document
will use "kerberos" as t
|`druid.auth.authenticator.kerberos.serverPrincipal`|`HTTP/_h...@example.com`|
SPNego service principal used by druid processes|empty|Yes|
|`druid.auth.authenticator.kerberos.serverKeytab`|`/etc/security/keytabs/spnego.service.keytab`|SPNego
service keytab used by druid processes|empty|Yes|
|`druid.auth.authenticator.kerberos.authToLocal`|`RULE:[1:$1@$0](dr...@example.com)s/.*/druid
DEFAULT`|It allows you to set a general rule for mapping principal names to
local user names. It will be used if there is not an explicit mapping for the
principal name that is being translated.|DEFAULT|No|
-|`druid.auth.authenticator.kerberos.excludedPaths`|`['/status','/health']`|
Array of HTTP paths which which does NOT need to be authenticated.|None|No|
|`druid.auth.authenticator.kerberos.cookieSignatureSecret`|`secretString`|
Secret used to sign authentication cookies. It is advisable to explicitly set
it, if you have multiple druid ndoes running on same machine with different
ports as the Cookie Specification does not guarantee isolation by port.||No|
|`druid.auth.authenticator.kerberos.authorizerName`|Depends on available
authorizers|Authorizer that requests should be directed to|Empty|Yes|
As a note, it is required that the SPNego principal in use by the druid
processes must start with HTTP (This specified by
[RFC-4559](https://tools.ietf.org/html/rfc4559)) and must be of the form
"HTTP/_HOST@REALM".
The special string _HOST will be replaced automatically with the value of
config `druid.host`
+### `druid.auth.authenticator.kerberos.excludedPaths`
+
+In older releases, the Kerberos authenticator had an `excludedPaths` property
that allowed the user to specify a list of paths where authentication checks
should be skipped. This property has been removed from the Kerberos
authenticator because the path exclusion functionality is now handled across
all authenticators/authorizers by setting `druid.auth.unsecuredPaths`, as
described in the [main auth documentation](../../design/auth.html).
+
### Auth to Local Syntax
`druid.auth.authenticator.kerberos.authToLocal` allows you to set a general
rules for mapping principal names to local user names.
The syntax for mapping rules is
`RULE:\[n:string](regexp)s/pattern/replacement/g`. The integer n indicates how
many components the target principal should have. If this matches, then a
string will be formed from string, substituting the realm of the principal for
$0 and the n‘th component of the principal for $n. e.g. if the principal was
druid/admin then `\[2:$2$1suffix]` would result in the string
`admindruidsuffix`.
diff --git
a/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
b/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
index 801d394..a58c799 100644
---
a/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
+++
b/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
@@ -66,19 +66,16 @@ import java.io.IOException;
import java.net.HttpCookie;
import java.security.Principal;
import java.text.SimpleDateFormat;
-import java.util.Collections;
import java.util.Date;
import java.util.EnumSet;
import java.util.HashMap;
import java.util.HashSet;
-import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.TimeZone;
import java.util.concurrent.ThreadLocalRandom;
-import java.util.regex.Pattern;
import java.util.stream.Collectors;
@@ -86,15 +83,11 @@ import java.util.stream.Collectors;
public class KerberosAuthenticator implements Authenticator
{
private static final Logger log = new
[GitHub] [incubator-druid] fjy merged pull request #7745: Remove obsolete isExcluded config from Kerberos authenticator
fjy merged pull request #7745: Remove obsolete isExcluded config from Kerberos authenticator URL: https://github.com/apache/incubator-druid/pull/7745 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: use memoized supplier for lazy singleton in SeekableStreamIndexTask.java (#7740)
This is an automated email from the ASF dual-hosted git repository.
fjy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new daf20b4 use memoized supplier for lazy singleton in
SeekableStreamIndexTask.java (#7740)
daf20b4 is described below
commit daf20b4b864bd6618150d6a7b2185b5677e8c406
Author: Himanshu
AuthorDate: Thu May 23 15:58:38 2019 -0700
use memoized supplier for lazy singleton in SeekableStreamIndexTask.java
(#7740)
---
.../indexing/seekablestream/SeekableStreamIndexTask.java | 16 +---
1 file changed, 5 insertions(+), 11 deletions(-)
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTask.java
b/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTask.java
index 37d472b..11ed7d4 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTask.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTask.java
@@ -24,6 +24,8 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Joiner;
import com.google.common.base.Optional;
import com.google.common.base.Preconditions;
+import com.google.common.base.Supplier;
+import com.google.common.base.Suppliers;
import org.apache.druid.data.input.InputRow;
import org.apache.druid.indexer.TaskStatus;
import org.apache.druid.indexing.appenderator.ActionBasedSegmentAllocator;
@@ -77,8 +79,7 @@ public abstract class
SeekableStreamIndexTaskhttps://github.com/apache/incubator-druid/issues/7724 for issues that
can cause.
// By the way, lazily init is synchronized because the runner may be needed
in multiple threads.
- private final Object runnerInitLock = new Object();
- private volatile SeekableStreamIndexTaskRunner runner;
+ private final Supplier> runnerSupplier;
public SeekableStreamIndexTask(
final String id,
@@ -112,6 +113,7 @@ public abstract class
SeekableStreamIndexTask
getRunner()
{
-if (runner == null) {
- synchronized (runnerInitLock) {
-if (runner == null) {
- runner = createTaskRunner();
-}
- }
-}
-
-return runner;
+return runnerSupplier.get();
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] fjy merged pull request #7740: use memoized supplier for lazy singleton in SeekableStreamIndexTask.java
fjy merged pull request #7740: use memoized supplier for lazy singleton in SeekableStreamIndexTask.java URL: https://github.com/apache/incubator-druid/pull/7740 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7742: [Backport] OrcStructConverter fix
jihoonson commented on issue #7742: [Backport] OrcStructConverter fix URL: https://github.com/apache/incubator-druid/pull/7742#issuecomment-495405097 Please remove the branch once this PR is merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson opened a new pull request #7747: Remove keepSegmentGranularity option from compaction
jihoonson opened a new pull request #7747: Remove keepSegmentGranularity option from compaction URL: https://github.com/apache/incubator-druid/pull/7747 `keepSegmentGranularity` option ignores the interval boundary of segments before compaction. This option is not much useful and was deprecated in 0.14.0 (https://github.com/apache/incubator-druid/pull/6758). People still want to use this option can set `segmentGranularity = ALL`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei opened a new pull request #7746: Fix dead code in IndexTask.collectIntervalsAndShardSpecs()
jon-wei opened a new pull request #7746: Fix dead code in IndexTask.collectIntervalsAndShardSpecs() URL: https://github.com/apache/incubator-druid/pull/7746 Fixes #7737 PR #5418 added meters for ingestion row counts, the logs adjusted in this PR referred to the old row ingestion count variables and weren't hooked up to the new meters. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7703: Web console: add log tailing to task log view
vogievetsky commented on a change in pull request #7703: Web console: add log
tailing to task log view
URL: https://github.com/apache/incubator-druid/pull/7703#discussion_r287138069
##
File path: web-console/src/components/show-log/show-log.tsx
##
@@ -71,12 +74,39 @@ export class ShowLog extends React.Component {
}
}
+ async tail() {
+ await this.getLogInfo();
+ if (this.state.tail) {
+ if (this.log.current) {
+this.log.current.scrollTo(0, this.log.current.scrollHeight);
+ }
+ setTimeout(() => {
+this.tail();
+ }, 2000);
+}
+}
+
+ private handleCheckboxChange = () => {
+this.setState({
+ tail: !this.state.tail
+});
+if (!this.state.tail) {
+ this.tail();
+}
+ }
+
+
render() {
const { endpoint, downloadFilename } = this.props;
const { logValue } = this.state;
return
+
[GitHub] [incubator-druid] jon-wei opened a new pull request #7745: Remove obsolete isExcluded config from Kerberos authenticator
jon-wei opened a new pull request #7745: Remove obsolete isExcluded config from Kerberos authenticator URL: https://github.com/apache/incubator-druid/pull/7745 `druid.auth.authenticator.kerberos.excludedPaths` is a configuration on the Kerberos authenticator that allowed the user to specify paths where authentication checks should be skipped. This config specific to Kerberos authenticator was superseded by `druid.auth.unsecuredPaths` which applies across all authenticators/authorizers. This PR removes the `isExcluded` property from the Kerberos authenticator. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287132991 ## File path: web-console/src/console-application.tsx ## @@ -101,6 +101,7 @@ export class ConsoleApplication extends React.Component
[GitHub] [incubator-druid] clintropolis opened a new pull request #7744: [Backport] fix AggregatorFactory.finalizeComputation implementations to be ok with null inputs
clintropolis opened a new pull request #7744: [Backport] fix AggregatorFactory.finalizeComputation implementations to be ok with null inputs URL: https://github.com/apache/incubator-druid/pull/7744 Backport of #7731 to 0.15.0-incubating. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287133909
##
File path: web-console/src/views/load-data-view/load-data-view.tsx
##
@@ -2348,13 +2355,22 @@ export class LoadDataView extends
React.Component => {
+try {
+ console.log(endpoint);
Review comment:
log
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287133661 ## File path: web-console/src/views/load-data-view/load-data-view.tsx ## @@ -281,9 +281,18 @@ export class LoadDataView extends React.Component
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287132688
##
File path: web-console/src/components/header-bar/header-bar.tsx
##
@@ -52,7 +52,7 @@ export type HeaderActiveTab = null | 'load-data' | 'query' |
'datasources' | 'se
export interface HeaderBarProps extends React.Props {
active: HeaderActiveTab;
hideLegacy: boolean;
- goToLoadDataView: () => void;
+ goToLoadDataView: (taskId: string) => void;
Review comment:
is this even needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on a change in pull request #7705: Web-Console: add go to editor button to tasks and supervisors view
vogievetsky commented on a change in pull request #7705: Web-Console: add go to
editor button to tasks and supervisors view
URL: https://github.com/apache/incubator-druid/pull/7705#discussion_r287134431
##
File path: web-console/src/views/load-data-view/load-data-view.tsx
##
@@ -2348,13 +2355,22 @@ export class LoadDataView extends
React.Component => {
+try {
+ console.log(endpoint);
+ const resp = await axios.get(endpoint);
+ let data = resp.data;
+ if (endpoint.indexOf('task') >= 0) data = resp.data.payload.spec;
+ this.updateSpec(data);
+} catch (e) {
+}
+ }
renderJsonSpecStage() {
const { goToTask } = this.props;
const { spec } = this.state;
-
return <>
-
+
Review comment:
typo?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] vogievetsky commented on issue #7723: Web-console: add resizable split screen layout to tasks and servers views
vogievetsky commented on issue #7723: Web-console: add resizable split screen layout to tasks and servers views URL: https://github.com/apache/incubator-druid/pull/7723#issuecomment-495380923 Could you please make the splitter match the style that it had prior to this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis opened a new pull request #7743: [Backport] allow quantiles merge aggregator to also accept doubles
clintropolis opened a new pull request #7743: [Backport] allow quantiles merge aggregator to also accept doubles URL: https://github.com/apache/incubator-druid/pull/7743 Backport of #7718 to 0.15.0-incubating. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch clintropolis-patch-1 created (now 3100c61)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch clintropolis-patch-1 in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. at 3100c61 [Backport] OrcStructConvert fix This branch includes the following new commits: new 3100c61 [Backport] OrcStructConvert fix The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] 01/01: [Backport] OrcStructConvert fix
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a commit to branch clintropolis-patch-1 in repository https://gitbox.apache.org/repos/asf/incubator-druid.git commit 3100c61f1220f6593d160818719a416e6ca53064 Author: Clint Wylie AuthorDate: Thu May 23 13:40:59 2019 -0700 [Backport] OrcStructConvert fix from #7738, this is a performance issue --- .../main/java/org/apache/druid/data/input/orc/OrcStructConverter.java | 2 -- 1 file changed, 2 deletions(-) diff --git a/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java b/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java index 20fbf06..69d113d 100644 --- a/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java +++ b/extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java @@ -166,8 +166,6 @@ public class OrcStructConverter fieldIndexCache.put(fields.get(i), i); } } -WritableComparable wc = struct.getFieldValue(fieldName); - int fieldIndex = fieldIndexCache.getOrDefault(fieldName, -1); return convertField(struct, fieldIndex); - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis opened a new pull request #7742: [Backport] OrcStructConvert fix
clintropolis opened a new pull request #7742: [Backport] OrcStructConvert fix URL: https://github.com/apache/incubator-druid/pull/7742 from #7738, this is a performance issue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #7138: 'core' ORC extension
clintropolis commented on a change in pull request #7138: 'core' ORC extension
URL: https://github.com/apache/incubator-druid/pull/7138#discussion_r287124475
##
File path:
extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java
##
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input.orc;
+
+import it.unimi.dsi.fastutil.objects.Object2IntMap;
+import it.unimi.dsi.fastutil.objects.Object2IntOpenHashMap;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.hadoop.hive.serde2.io.DateWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.io.BooleanWritable;
+import org.apache.hadoop.io.ByteWritable;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.io.DoubleWritable;
+import org.apache.hadoop.io.FloatWritable;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.io.ShortWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.orc.TypeDescription;
+import org.apache.orc.mapred.OrcList;
+import org.apache.orc.mapred.OrcMap;
+import org.apache.orc.mapred.OrcStruct;
+import org.apache.orc.mapred.OrcTimestamp;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class OrcStructConverter
+{
+ @Nonnull
+ private static List convertList(TypeDescription fieldDescription,
OrcList orcList, boolean binaryAsString)
+ {
+// if primitive list, convert primitives
+TypeDescription listType = fieldDescription.getChildren().get(0);
+if (listType.getCategory().isPrimitive()) {
+ return (List) orcList.stream()
+ .map(li -> convertPrimitive(listType,
(WritableComparable) li, binaryAsString))
+ .collect(Collectors.toList());
+}
+return new ArrayList(orcList);
+ }
+
+ private static Map convertMap(
+ TypeDescription fieldDescription,
+ OrcMap map,
+ boolean binaryAsString
+ )
+ {
+Map converted = new HashMap<>();
+TypeDescription keyDescription = fieldDescription.getChildren().get(0);
+TypeDescription valueDescription = fieldDescription.getChildren().get(1);
+for (WritableComparable key : map.navigableKeySet()) {
+ Object newKey = convertPrimitive(keyDescription, key, binaryAsString);
+ if (valueDescription.getCategory().isPrimitive()) {
+converted.put(newKey, convertPrimitive(valueDescription, map.get(key),
binaryAsString));
+ } else {
+converted.put(newKey, map.get(key));
+ }
+}
+return converted;
+ }
+
+ private static Object convertPrimitive(TypeDescription fieldDescription,
WritableComparable field, boolean binaryAsString)
+ {
+/*
+ORC TYPEWRITABLE TYPE
+binary org.apache.hadoop.io.BytesWritable
+bigint org.apache.hadoop.io.LongWritable
+boolean org.apache.hadoop.io.BooleanWritable
+charorg.apache.hadoop.io.Text
+dateorg.apache.hadoop.hive.serde2.io.DateWritable
+decimal org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
+double org.apache.hadoop.io.DoubleWritable
+float org.apache.hadoop.io.FloatWritable
+int org.apache.hadoop.io.IntWritable
+smallintorg.apache.hadoop.io.ShortWritable
+string org.apache.hadoop.io.Text
+timestamp org.apache.orc.mapred.OrcTimestamp
+tinyint org.apache.hadoop.io.ByteWritable
+varchar org.apache.hadoop.io.Text
+ */
+switch (fieldDescription.getCategory()) {
+ case STRING:
+ case CHAR:
+ case VARCHAR:
+return ((Text) field).toString();
+ case BOOLEAN:
+return ((BooleanWritable) field).get();
+ case BYTE:
+return ((ByteWritable) field).get();
+ case SHORT:
+
[incubator-druid] branch master updated: Remove unnecessary principal handling in KerberosAuthenticator (#7685)
This is an automated email from the ASF dual-hosted git repository.
jonwei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 54b3f36 Remove unnecessary principal handling in
KerberosAuthenticator (#7685)
54b3f36 is described below
commit 54b3f363c4fc73fcc47b757bda8071a877b967f9
Author: Jonathan Wei
AuthorDate: Thu May 23 13:15:44 2019 -0700
Remove unnecessary principal handling in KerberosAuthenticator (#7685)
---
.../security/kerberos/KerberosAuthenticator.java | 125 +++--
1 file changed, 13 insertions(+), 112 deletions(-)
diff --git
a/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
b/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
index d7edfaa..801d394 100644
---
a/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
+++
b/extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
@@ -43,20 +43,9 @@ import
org.apache.hadoop.security.authentication.util.SignerException;
import org.apache.hadoop.security.authentication.util.SignerSecretProvider;
import org.eclipse.jetty.client.api.Request;
import org.eclipse.jetty.http.HttpHeader;
-import sun.security.krb5.EncryptedData;
-import sun.security.krb5.EncryptionKey;
-import sun.security.krb5.internal.APReq;
-import sun.security.krb5.internal.EncTicketPart;
-import sun.security.krb5.internal.Krb5;
-import sun.security.krb5.internal.Ticket;
-import sun.security.krb5.internal.crypto.KeyUsage;
-import sun.security.util.DerInputStream;
-import sun.security.util.DerValue;
import javax.security.auth.Subject;
-import javax.security.auth.kerberos.KerberosKey;
import javax.security.auth.kerberos.KerberosPrincipal;
-import javax.security.auth.kerberos.KeyTab;
import javax.security.auth.login.AppConfigurationEntry;
import javax.security.auth.login.Configuration;
import javax.security.auth.login.LoginContext;
@@ -89,7 +78,6 @@ import java.util.Properties;
import java.util.Set;
import java.util.TimeZone;
import java.util.concurrent.ThreadLocalRandom;
-import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
@@ -230,56 +218,36 @@ public class KerberosAuthenticator implements
Authenticator
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain filterChain)
throws IOException, ServletException
{
-HttpServletRequest httpReq = (HttpServletRequest) request;
-
// If there's already an auth result, then we have authenticated
already, skip this.
if (request.getAttribute(AuthConfig.DRUID_AUTHENTICATION_RESULT) !=
null) {
filterChain.doFilter(request, response);
return;
}
+// In the hadoop-auth 2.7.3 code that this was adapted from, the login
would've occurred during init() of
+// the AuthenticationFilter via
`initializeAuthHandler(authHandlerClassName, filterConfig)`.
+// Since we co-exist with other authentication schemes, don't login
until we've checked that
+// some other Authenticator didn't already validate this request.
if (loginContext == null) {
initializeKerberosLogin();
}
+// Checking for excluded paths is Druid-specific, not from hadoop-auth
String path = ((HttpServletRequest) request).getRequestURI();
if (isExcluded(path)) {
filterChain.doFilter(request, response);
} else {
- String clientPrincipal;
- try {
-Cookie[] cookies = httpReq.getCookies();
-if (cookies == null) {
- clientPrincipal =
getPrincipalFromRequestNew((HttpServletRequest) request);
-} else {
- clientPrincipal = null;
- for (Cookie cookie : cookies) {
-if ("hadoop.auth".equals(cookie.getName())) {
- Matcher matcher =
HADOOP_AUTH_COOKIE_REGEX.matcher(cookie.getValue());
- if (matcher.matches()) {
-clientPrincipal = matcher.group(1);
-break;
- }
-}
- }
-}
- }
- catch (Exception ex) {
-clientPrincipal = null;
- }
-
- if (clientPrincipal != null) {
-request.setAttribute(
-AuthConfig.DRUID_AUTHENTICATION_RESULT,
-new AuthenticationResult(clientPrincipal, authorizerName,
name, null)
-);
- }
+ // Run the original doFilter method, but with modifications to error
handling
+ doFilterSuper(request, response, filterChain);
}
-
-doFilterSuper(request, response,
[GitHub] [incubator-druid] jon-wei merged pull request #7685: Remove unnecessary principal handling in KerberosAuthenticator
jon-wei merged pull request #7685: Remove unnecessary principal handling in KerberosAuthenticator URL: https://github.com/apache/incubator-druid/pull/7685 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei closed issue #7527: druid-kerberos module uses internal Java APIs removed in Java 9 and above
jon-wei closed issue #7527: druid-kerberos module uses internal Java APIs removed in Java 9 and above URL: https://github.com/apache/incubator-druid/issues/7527 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #7685: Remove unnecessary principal handling in KerberosAuthenticator
jon-wei commented on a change in pull request #7685: Remove unnecessary
principal handling in KerberosAuthenticator
URL: https://github.com/apache/incubator-druid/pull/7685#discussion_r287116248
##
File path:
extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
##
@@ -230,56 +218,36 @@ protected AuthenticationToken
getToken(HttpServletRequest request) throws Authen
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain filterChain)
throws IOException, ServletException
{
-HttpServletRequest httpReq = (HttpServletRequest) request;
-
// If there's already an auth result, then we have authenticated
already, skip this.
if (request.getAttribute(AuthConfig.DRUID_AUTHENTICATION_RESULT) !=
null) {
filterChain.doFilter(request, response);
return;
}
+// In the hadoop-auth 2.7.3 code that this was adapted from, the login
would've occurred during init() of
+// the AuthenticationFilter via
`initializeAuthHandler(authHandlerClassName, filterConfig)`.
+// Since we co-exist with other authentication schemes, don't login
until we've checked that
+// some other Authenticator didn't already validate this request.
if (loginContext == null) {
initializeKerberosLogin();
}
+// Checking for excluded paths is Druid-specific, not from hadoop-auth
String path = ((HttpServletRequest) request).getRequestURI();
if (isExcluded(path)) {
filterChain.doFilter(request, response);
} else {
- String clientPrincipal;
- try {
-Cookie[] cookies = httpReq.getCookies();
-if (cookies == null) {
- clientPrincipal =
getPrincipalFromRequestNew((HttpServletRequest) request);
-} else {
- clientPrincipal = null;
- for (Cookie cookie : cookies) {
-if ("hadoop.auth".equals(cookie.getName())) {
- Matcher matcher =
HADOOP_AUTH_COOKIE_REGEX.matcher(cookie.getValue());
- if (matcher.matches()) {
-clientPrincipal = matcher.group(1);
-break;
- }
-}
- }
-}
- }
- catch (Exception ex) {
-clientPrincipal = null;
- }
-
- if (clientPrincipal != null) {
-request.setAttribute(
-AuthConfig.DRUID_AUTHENTICATION_RESULT,
-new AuthenticationResult(clientPrincipal, authorizerName,
name, null)
-);
- }
+ // Run the original doFilter method, but with modifications to error
handling
+ doFilterSuper(request, response, filterChain);
}
-
-doFilterSuper(request, response, filterChain);
Review comment:
I'll raise a follow-on PR to address that, will go ahead and merge this so
that the JDK9+ compat is unblocked.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7729: SeekableStreamIndexTaskRunner: Lazy init of runner.
jihoonson commented on issue #7729: SeekableStreamIndexTaskRunner: Lazy init of runner. URL: https://github.com/apache/incubator-druid/pull/7729#issuecomment-495367228 To me, this bug doesn't look critical enough to backport to 0.15.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #7685: Remove unnecessary principal handling in KerberosAuthenticator
jon-wei commented on a change in pull request #7685: Remove unnecessary
principal handling in KerberosAuthenticator
URL: https://github.com/apache/incubator-druid/pull/7685#discussion_r287115003
##
File path:
extensions-core/druid-kerberos/src/main/java/org/apache/druid/security/kerberos/KerberosAuthenticator.java
##
@@ -230,56 +218,36 @@ protected AuthenticationToken
getToken(HttpServletRequest request) throws Authen
public void doFilter(ServletRequest request, ServletResponse response,
FilterChain filterChain)
throws IOException, ServletException
{
-HttpServletRequest httpReq = (HttpServletRequest) request;
-
// If there's already an auth result, then we have authenticated
already, skip this.
if (request.getAttribute(AuthConfig.DRUID_AUTHENTICATION_RESULT) !=
null) {
filterChain.doFilter(request, response);
return;
}
+// In the hadoop-auth 2.7.3 code that this was adapted from, the login
would've occurred during init() of
+// the AuthenticationFilter via
`initializeAuthHandler(authHandlerClassName, filterConfig)`.
+// Since we co-exist with other authentication schemes, don't login
until we've checked that
+// some other Authenticator didn't already validate this request.
if (loginContext == null) {
initializeKerberosLogin();
}
+// Checking for excluded paths is Druid-specific, not from hadoop-auth
String path = ((HttpServletRequest) request).getRequestURI();
if (isExcluded(path)) {
filterChain.doFilter(request, response);
} else {
- String clientPrincipal;
- try {
-Cookie[] cookies = httpReq.getCookies();
-if (cookies == null) {
- clientPrincipal =
getPrincipalFromRequestNew((HttpServletRequest) request);
-} else {
- clientPrincipal = null;
- for (Cookie cookie : cookies) {
-if ("hadoop.auth".equals(cookie.getName())) {
- Matcher matcher =
HADOOP_AUTH_COOKIE_REGEX.matcher(cookie.getValue());
- if (matcher.matches()) {
-clientPrincipal = matcher.group(1);
-break;
- }
-}
- }
-}
- }
- catch (Exception ex) {
-clientPrincipal = null;
- }
-
- if (clientPrincipal != null) {
-request.setAttribute(
-AuthConfig.DRUID_AUTHENTICATION_RESULT,
-new AuthenticationResult(clientPrincipal, authorizerName,
name, null)
-);
- }
+ // Run the original doFilter method, but with modifications to error
handling
+ doFilterSuper(request, response, filterChain);
}
-
-doFilterSuper(request, response, filterChain);
Review comment:
Hm, this `isExcluded` parameter should actually be removed, it was part of
the original Kerberos authentication extension before the larger auth system
was added.
This path exclusion functionality should now be handled by setting
`druid.auth.unsecuredPaths`, a filter at the start of the filterChain would
authenticate+authorize requests made to paths specified in this config.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7741: Reindexing tasks containing thetaSketches resulting in incorrect sketch values
jihoonson commented on issue #7741: Reindexing tasks containing thetaSketches resulting in incorrect sketch values URL: https://github.com/apache/incubator-druid/issues/7741#issuecomment-495364906 I see some differences between two specs in the size of thetaSketch and partitionSpec. The thetaSketch gives an approximate result, and so I guess maybe the different result comes from the mismatch in size and the partitionSpec which is expected? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] a2l007 opened a new issue #7741: Reindexing tasks containing thetaSketches resulting in incorrect sketch values
a2l007 opened a new issue #7741: Reindexing tasks containing thetaSketches
resulting in incorrect sketch values
URL: https://github.com/apache/incubator-druid/issues/7741
### Affected Version
0.13.0-incubating
### Description
One of our clusters running on 0.13.0 is seeing data corruption issues with
thetasketches on reindexing. The basefact datasource was indexed with
`maxBytesInMemory` set as -1 due to #6743 . But the reindexed slices off of
this basefact has incorrect values for thetasketch columns.
**Ingestion Spec for basefact datasource:**
```
{
"type" : "index_hadoop",
"id" : "index_base",
"spec" : {
"dataSchema" : {
"dataSource" : "basefact",
"parser" : {
"type" : "avro_hadoop",
"parseSpec" : {
"format" : "avro",
"timestampSpec" : {
"column" : "date",
"format" : "MMdd"
},
"dimensionsSpec" : {
"dimensions" : [ "src_pty_id" ]
}
}
},
"metricsSpec" : [ {
"type" : "thetaSketch",
"name" : "test_sketch",
"fieldName" : "test_sketch",
"size" : 1048576,
"shouldFinalize" : true,
"isInputThetaSketch" : true,
"errorBoundsStdDev" : null
} ],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "DAY",
"queryGranularity" : {
"type" : "none"
},
"rollup" : true,
"intervals" : [ "2019-05-22T00:00:00.000Z/2019-05-23T00:00:00.000Z" ]
},
"transformSpec" : {
"filter" : null,
"transforms" : [ ]
}
},
"ioConfig" : {
"type" : "hadoop",
"inputSpec" : {
"type" : "static",
"paths" : "///projects/indexData",
"inputFormat" :
"org.apache.druid.data.input.avro.AvroValueInputFormat"
},
"metadataUpdateSpec" : null,
"segmentOutputPath" : null
},
"tuningConfig" : {
"type" : "hadoop",
"workingPath" : null,
"version" : "2019-05-23T08:33:19.990Z",
"partitionsSpec" : {
"type" : "hashed",
"targetPartitionSize" : 105,
"maxPartitionSize" : 1575000,
"assumeGrouped" : true,
"numShards" : -1,
"partitionDimensions" : [ ]
},
"shardSpecs" : { },
"indexSpec" : {
"bitmap" : {
"type" : "concise"
},
"dimensionCompression" : "lz4",
"metricCompression" : "lz4",
"longEncoding" : "longs"
},
"maxRowsInMemory" : 15,
"maxBytesInMemory" : -1,
"leaveIntermediate" : false,
"cleanupOnFailure" : true,
"overwriteFiles" : false,
"ignoreInvalidRows" : false,
"jobProperties" : {
"fs.permissions.umask-mode" : "027"
},
"combineText" : false,
"useCombiner" : false,
"buildV9Directly" : true,
"numBackgroundPersistThreads" : 0,
"forceExtendableShardSpecs" : false,
"useExplicitVersion" : false,
"allowedHadoopPrefix" : [ ],
"logParseExceptions" : false,
"maxParseExceptions" : 0
}
},
"hadoopDependencyCoordinates" : null,
"classpathPrefix" : null,
"context" : { },
"groupId" : "index_hadoop_basefact_2019-05-23T08:33:19.990Z",
"dataSource" : "basefact",
"resource" : {
"availabilityGroup" : "index_hadoop_basefact_2019-05-23T08:33:19.990Z",
"requiredCapacity" : 1
}
}
```
**Ingestion spec for slice datasource:**
```
{
"type" : "index_hadoop",
"id" : "index_slice",
"spec" : {
"dataSchema" : {
"dataSource" : "slice",
"parser" : {
"type" : "avro_hadoop",
"parseSpec" : {
"format" : "avro",
"timestampSpec" : {
"column" : "date",
"format" : "MMdd"
},
"dimensionsSpec" : {
"dimensions" : [ "src_pty_id" ]
}
}
},
"metricsSpec" : [ {
"type" : "thetaSketch",
"name" : "test_sketch",
"fieldName" : "test_sketch",
"size" : 131072,
"shouldFinalize" : true,
"isInputThetaSketch" : true,
"errorBoundsStdDev" : null
} ],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "DAY",
"queryGranularity" : {
"type" : "none"
},
"rollup" : true,
"intervals" : [ "2019-05-22T00:00:00.000Z/2019-05-23T00:00:00.000Z" ]
},
"transformSpec" : {
"filter" : null,
[GitHub] [incubator-druid] surekhasaharan commented on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status
surekhasaharan commented on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status URL: https://github.com/apache/incubator-druid/issues/7571#issuecomment-495357873 @leventov @jihoonson @gianm thanks for the discussion, I will work on this PR tomorrow and hopefully we can get it closed before 0.15. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #7736: Druid keep using segment from the segment-cache the one not exists on deep storage anymore
himanshug commented on issue #7736: Druid keep using segment from the segment-cache the one not exists on deep storage anymore URL: https://github.com/apache/incubator-druid/issues/7736#issuecomment-495353746 sounds like expected behavior: it takes time for coordinator to notice removal of a segment in metadataStore and then asking the historical node to drop a segment, it should go away after a while. if it never goes away then there is something fishy going on. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7738: Fix some problems reported by PVS-Studio
gianm commented on a change in pull request #7738: Fix some problems reported
by PVS-Studio
URL: https://github.com/apache/incubator-druid/pull/7738#discussion_r287071695
##
File path:
extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java
##
@@ -52,7 +52,7 @@ public void kill(DataSegment segment) throws
SegmentLoadingException
Map loadSpec = segment.getLoadSpec();
final String bucket = MapUtils.getString(loadSpec, "bucket");
final String indexPath = MapUtils.getString(loadSpec, "path");
-final String descriptorPath = indexPath.substring(0,
indexPath.lastIndexOf('/')) + "/descriptor.json";
+final String descriptorPath = indexPath.substring(0,
indexPath.lastIndexOf('/')) + "/descriptor.json"; //-V6009
Review comment:
What does `-V6009` mean?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7738: Fix some problems reported by PVS-Studio
gianm commented on a change in pull request #7738: Fix some problems reported by PVS-Studio URL: https://github.com/apache/incubator-druid/pull/7738#discussion_r287070364 ## File path: extensions-contrib/kafka-eight-simpleConsumer/src/main/java/org/apache/druid/firehose/kafka/KafkaSimpleConsumer.java ## @@ -274,7 +272,7 @@ private long getOffset(boolean earliest) throws InterruptedException } } -return response != null ? filterAndDecode(response.messageSet(topic, partitionId), offset) : EMPTY_MSGS; +return filterAndDecode(response.messageSet(topic, partitionId), offset); Review comment: Is it impossible for `response` to be null at this point? I guess the idea is that if it's null, `errorCode` should be `ErrorMapping.UnknownCode()`, and therefore `needNewLeader` should be true, and so we should hit the `continue;` and keep looping. The logic is a bit nontrivial to follow, so if that is accurate, a comment or `Preconditions.checkState` would be helpful. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #7738: Fix some problems reported by PVS-Studio
gianm commented on a change in pull request #7738: Fix some problems reported
by PVS-Studio
URL: https://github.com/apache/incubator-druid/pull/7738#discussion_r287071863
##
File path:
extensions-core/histogram/src/main/java/org/apache/druid/query/aggregation/histogram/ApproximateHistogram.java
##
@@ -419,7 +419,7 @@ protected void mergeInsert(final int mergeAt, int
insertAt, final float v, final
// use unused slot to shift array left or right and make space for the
new bin to insert
if (insertAt < unusedIndex) {
shiftRight(insertAt, unusedIndex);
- } else if (insertAt >= unusedIndex) {
+ } else { // insertAt >= unusedIndex
Review comment:
How about turning this into an `assert` instead of a comment (or just delete
it, it's pretty obvious just from reading the code).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #7729: SeekableStreamIndexTaskRunner: Lazy init of runner.
himanshug commented on a change in pull request #7729:
SeekableStreamIndexTaskRunner: Lazy init of runner.
URL: https://github.com/apache/incubator-druid/pull/7729#discussion_r287083342
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTask.java
##
@@ -283,13 +282,20 @@ public boolean withinMinMaxRecordTime(final InputRow row)
@VisibleForTesting
public Appenderator getAppenderator()
{
-return runner.getAppenderator();
+return getRunner().getAppenderator();
}
@VisibleForTesting
public SeekableStreamIndexTaskRunner
getRunner()
{
+if (runner == null) {
+ synchronized (runnerInitLock) {
+if (runner == null) {
+ runner = createTaskRunner();
+}
+ }
+}
Review comment:
https://github.com/apache/incubator-druid/pull/7740
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug opened a new pull request #7740: use memoized supplier for lazy singleton in SeekableStreamIndexTask.java
himanshug opened a new pull request #7740: use memoized supplier for lazy singleton in SeekableStreamIndexTask.java URL: https://github.com/apache/incubator-druid/pull/7740 https://github.com/apache/incubator-druid/pull/7729#discussion_r286661510 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #7729: SeekableStreamIndexTaskRunner: Lazy init of runner.
himanshug commented on a change in pull request #7729:
SeekableStreamIndexTaskRunner: Lazy init of runner.
URL: https://github.com/apache/incubator-druid/pull/7729#discussion_r287083036
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTask.java
##
@@ -283,13 +282,20 @@ public boolean withinMinMaxRecordTime(final InputRow row)
@VisibleForTesting
public Appenderator getAppenderator()
{
-return runner.getAppenderator();
+return getRunner().getAppenderator();
}
@VisibleForTesting
public SeekableStreamIndexTaskRunner
getRunner()
{
+if (runner == null) {
+ synchronized (runnerInitLock) {
+if (runner == null) {
+ runner = createTaskRunner();
+}
+ }
+}
Review comment:
that sounds nice, now I remember seeing that :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #7716: AggregatorUtil should cache parsed expression to avoid memory problem (OOM/FGC) when Expression is used in metricsSpec
himanshug commented on a change in pull request #7716: AggregatorUtil should
cache parsed expression to avoid memory problem (OOM/FGC) when Expression is
used in metricsSpec
URL: https://github.com/apache/incubator-druid/pull/7716#discussion_r287079210
##
File path:
processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
##
@@ -196,7 +246,7 @@ static BaseFloatColumnValueSelector
makeColumnValueSelectorWithFloatDefault(
if (fieldName != null) {
return metricFactory.makeColumnValueSelector(fieldName);
} else {
- final Expr expr = Parser.parse(fieldExpression, macroTable);
+ final Expr expr = parseIfAbsent(fieldExpression, macroTable);
Review comment:
sent a PR to your branch, pls merge that. I meant to remove cache altogether
and tie lifetime of parsed expression to the object holding it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status
jihoonson commented on issue #7571: Optimize coordinator API to retrieve segments with overshadowed status URL: https://github.com/apache/incubator-druid/issues/7571#issuecomment-495331651 > Note that I unblocked #7425 only on the condition that people agree with the proposed approach and it's being implemented immediately: #7425 (comment). If I knew that new round of debate will start, I wouldn't unblock that PR and insisted on completely resolving that question before merging. @leventov thank you for additional context. I didn't notice this before. This sounds like Both https://github.com/apache/incubator-druid/pull/7425 and https://github.com/apache/incubator-druid/pull/7595 are supposed to be done in a single PR, but they are separate for the sake of convenience of the author and reviewers. And I like to avoid two potential problems in https://github.com/apache/incubator-druid/pull/7425, i.e., slow API performance and internal API mismatch between versions. So, I think it makes sense to add this to 0.15.0 as long as https://github.com/apache/incubator-druid/pull/7595 won't take long time. > So what about the following: > > - Immutable, but extensible DataSegment. > - Immutable SegmentWithOvershadowedStatus which extends DataSegment (note: they also have different JSON representations). > - Although immutable, both are decidedly containers, not "data classes". They both prohibit equals() and hashCode() as described here. Your suggestion sounds good to me. > Memory and concurrency (see #7595). https://github.com/apache/incubator-druid/pull/7595 looks about eager computation of overshadowed segments rather than mutable segments. Would you tell me more how the mutable segment helps concurrency? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #7718: allow quantiles merge aggregator to also accept doubles
himanshug merged pull request #7718: allow quantiles merge aggregator to also accept doubles URL: https://github.com/apache/incubator-druid/pull/7718 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug closed issue #7660: Quantiles sketch agg fails on inner query numeric post-agg columns
himanshug closed issue #7660: Quantiles sketch agg fails on inner query numeric post-agg columns URL: https://github.com/apache/incubator-druid/issues/7660 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: allow quantiles merge aggregator to also accept doubles (#7718)
This is an automated email from the ASF dual-hosted git repository.
himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 23e96d1 allow quantiles merge aggregator to also accept doubles
(#7718)
23e96d1 is described below
commit 23e96d15d41fef0cfe4d1815421b19fca3005334
Author: Clint Wylie
AuthorDate: Thu May 23 11:13:41 2019 -0700
allow quantiles merge aggregator to also accept doubles (#7718)
* allow quantiles merge aggregator to also accept doubles
* consolidate dupe
* import
---
.../quantiles/DoublesSketchBuildAggregator.java| 3 -
.../quantiles/DoublesSketchMergeAggregator.java| 23 +++--
.../DoublesSketchMergeBufferAggregator.java| 11 +--
.../sql/DoublesSketchSqlAggregatorTest.java| 106 +
4 files changed, 106 insertions(+), 37 deletions(-)
diff --git
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
index bd46fc5..18f94a9 100644
---
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
+++
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
@@ -28,14 +28,12 @@ public class DoublesSketchBuildAggregator implements
Aggregator
{
private final ColumnValueSelector valueSelector;
- private final int size;
private UpdateDoublesSketch sketch;
public DoublesSketchBuildAggregator(final ColumnValueSelector
valueSelector, final int size)
{
this.valueSelector = valueSelector;
-this.size = size;
sketch = DoublesSketch.builder().setK(size).build();
}
@@ -68,5 +66,4 @@ public class DoublesSketchBuildAggregator implements
Aggregator
{
sketch = null;
}
-
}
diff --git
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
index 325a6f2..4598048 100644
---
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
+++
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
@@ -27,10 +27,10 @@ import org.apache.druid.segment.ColumnValueSelector;
public class DoublesSketchMergeAggregator implements Aggregator
{
- private final ColumnValueSelector selector;
+ private final ColumnValueSelector selector;
private DoublesUnion union;
- public DoublesSketchMergeAggregator(final ColumnValueSelector
selector, final int k)
+ public DoublesSketchMergeAggregator(final ColumnValueSelector selector,
final int k)
{
this.selector = selector;
union = DoublesUnion.builder().setMaxK(k).build();
@@ -39,13 +39,10 @@ public class DoublesSketchMergeAggregator implements
Aggregator
@Override
public synchronized void aggregate()
{
-final DoublesSketch sketch = selector.getObject();
-if (sketch == null) {
- return;
-}
-union.update(sketch);
+updateUnion(selector, union);
}
+
@Override
public synchronized Object get()
{
@@ -70,4 +67,16 @@ public class DoublesSketchMergeAggregator implements
Aggregator
union = null;
}
+ static void updateUnion(ColumnValueSelector selector, DoublesUnion union)
+ {
+final Object object = selector.getObject();
+if (object == null) {
+ return;
+}
+if (object instanceof DoublesSketch) {
+ union.update((DoublesSketch) object);
+} else {
+ union.update(selector.getDouble());
+}
+ }
}
diff --git
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
index ffe9009..f5a1e9d 100644
---
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
+++
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
@@ -20,7 +20,6 @@
package org.apache.druid.query.aggregation.datasketches.quantiles;
import com.yahoo.memory.WritableMemory;
-import com.yahoo.sketches.quantiles.DoublesSketch;
import com.yahoo.sketches.quantiles.DoublesUnion;
import
[GitHub] [incubator-druid] himanshug merged pull request #7717: Adding influxdb emitter as a contrib extension
himanshug merged pull request #7717: Adding influxdb emitter as a contrib extension URL: https://github.com/apache/incubator-druid/pull/7717 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Adding influxdb emitter as a contrib extension (#7717)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 6964ac2 Adding influxdb emitter as a contrib extension (#7717) 6964ac2 is described below commit 6964ac23a2a3b95f29a2d4aa443ef9fcbc84dfdb Author: awelsh93 <32643586+awels...@users.noreply.github.com> AuthorDate: Thu May 23 19:11:48 2019 +0100 Adding influxdb emitter as a contrib extension (#7717) * Adding influxdb emitter as a contrib extension * addressing code review comments --- distribution/pom.xml | 2 + .../extensions-contrib/influxdb-emitter.md | 75 docs/content/development/extensions.md | 1 + extensions-contrib/influxdb-emitter/pom.xml| 74 +++ .../druid/emitter/influxdb/InfluxdbEmitter.java| 214 + .../emitter/influxdb/InfluxdbEmitterConfig.java| 196 +++ .../emitter/influxdb/InfluxdbEmitterModule.java| 61 ++ .../org.apache.druid.initialization.DruidModule| 16 ++ .../influxdb/InfluxdbEmitterConfigTest.java| 212 .../emitter/influxdb/InfluxdbEmitterTest.java | 208 pom.xml| 1 + 11 files changed, 1060 insertions(+) diff --git a/distribution/pom.xml b/distribution/pom.xml index e25b3c5..4ec9b26 100644 --- a/distribution/pom.xml +++ b/distribution/pom.xml @@ -324,6 +324,8 @@ -c org.apache.druid.extensions.contrib:druid-influx-extensions -c + org.apache.druid.extensions.contrib:druid-influxdb-emitter +-c org.apache.druid.extensions.contrib:druid-kafka-eight-simple-consumer -c org.apache.druid.extensions.contrib:kafka-emitter diff --git a/docs/content/development/extensions-contrib/influxdb-emitter.md b/docs/content/development/extensions-contrib/influxdb-emitter.md new file mode 100644 index 000..138a0bb --- /dev/null +++ b/docs/content/development/extensions-contrib/influxdb-emitter.md @@ -0,0 +1,75 @@ +--- +layout: doc_page +title: "InfluxDB Emitter" +--- + + + +# InfluxDB Emitter + +To use this Apache Druid (incubating) extension, make sure to [include](../../operations/including-extensions.html) `druid-influxdb-emitter` extension. + +## Introduction + +This extension emits druid metrics to [InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/) over HTTP. Currently this emitter only emits service metric events to InfluxDB (See [Druid metrics](../../operations/metrics.html) for a list of metrics). +When a metric event is fired it is added to a queue of events. After a configurable amount of time, the events on the queue are transformed to InfluxDB's line protocol +and POSTed to the InfluxDB HTTP API. The entire queue is flushed at this point. The queue is also flushed as the emitter is shutdown. + +Note that authentication and authorization must be [enabled](https://docs.influxdata.com/influxdb/v1.7/administration/authentication_and_authorization/) on the InfluxDB server. + +## Configuration + +All the configuration parameters for the influxdb emitter are under `druid.emitter.influxdb`. + +|Property|Description|Required?|Default| +||---|-|---| +|`druid.emitter.influxdb.hostname`|The hostname of the InfluxDB server.|Yes|N/A| +|`druid.emitter.influxdb.port`|The port of the InfluxDB server.|No|8086| +|`druid.emitter.influxdb.databaseName`|The name of the database in InfluxDB.|Yes|N/A| +|`druid.emitter.influxdb.maxQueueSize`|The size of the queue that holds events.|No|Integer.Max_Value(=2^31-1)| +|`druid.emitter.influxdb.flushPeriod`|How often (in milliseconds) the events queue is parsed into Line Protocol and POSTed to InfluxDB.|No|6| +|`druid.emitter.influxdb.flushDelay`|How long (in milliseconds) the scheduled method will wait until it first runs.|No|6| +|`druid.emitter.influxdb.influxdbUserName`|The username for authenticating with the InfluxDB database.|Yes|N/A| +|`druid.emitter.influxdb.influxdbPassword`|The password of the database authorized user|Yes|N/A| +|`druid.emitter.influxdb.dimensionWhitelist`|A whitelist of metric dimensions to include as tags|No|`["dataSource","type","numMetrics","numDimensions","threshold","dimension","taskType","taskStatus","tier"]`| + +## InfluxDB Line Protocol + +An example of how this emitter parses a Druid metric event into InfluxDB's [line protocol](https://docs.influxdata.com/influxdb/v1.7/write_protocols/line_protocol_reference/) is given
[GitHub] [incubator-druid] fjy merged pull request #7739: [Backport] reorganizing the ToC
fjy merged pull request #7739: [Backport] reorganizing the ToC URL: https://github.com/apache/incubator-druid/pull/7739 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch 0.15.0-incubating updated: reorganizing the ToC (#7734) (#7739)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch 0.15.0-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/0.15.0-incubating by this push: new 8bd5170 reorganizing the ToC (#7734) (#7739) 8bd5170 is described below commit 8bd5170e0a00b65d4f291baaeac91bfe893e8d1d Author: Jihoon Son AuthorDate: Thu May 23 10:41:23 2019 -0700 reorganizing the ToC (#7734) (#7739) --- docs/content/toc.md | 46 +- 1 file changed, 25 insertions(+), 21 deletions(-) diff --git a/docs/content/toc.md b/docs/content/toc.md index 2364470..57ed632 100644 --- a/docs/content/toc.md +++ b/docs/content/toc.md @@ -24,7 +24,7 @@ layout: toc ## Getting Started * [Design](/docs/VERSION/design/index.html) * [What is Druid?](/docs/VERSION/design/index.html#what-is-druid) -* [When should I use Druid](/docs/VERSION/design/index.html#when-to-use-druid) +* [When should I use Druid?](/docs/VERSION/design/index.html#when-to-use-druid) * [Architecture](/docs/VERSION/design/index.html#architecture) * [Datasources & Segments](/docs/VERSION/design/index.html#datasources-and-segments) * [Query processing](/docs/VERSION/design/index.html#query-processing) @@ -32,7 +32,7 @@ layout: toc * [Ingestion overview](/docs/VERSION/ingestion/index.html) * [Getting Started](/docs/VERSION/operations/getting-started.html) * [Single-server Quickstart](/docs/VERSION/tutorials/index.html) - * [Tutorial: Loading a file](/docs/VERSION/tutorials/tutorial-batch.html) + * [Tutorial: Loading a file from local disk](/docs/VERSION/tutorials/tutorial-batch.html) * [Tutorial: Loading stream data from Apache Kafka](/docs/VERSION/tutorials/tutorial-kafka.html) * [Tutorial: Loading a file using Apache Hadoop](/docs/VERSION/tutorials/tutorial-batch-hadoop.html) * [Tutorial: Loading stream data using HTTP push](/docs/VERSION/tutorials/tutorial-tranquility.html) @@ -44,8 +44,11 @@ layout: toc * [Tutorial: Compacting segments](/docs/VERSION/tutorials/tutorial-compaction.html) * [Tutorial: Deleting data](/docs/VERSION/tutorials/tutorial-delete-data.html) * [Tutorial: Writing your own ingestion specs](/docs/VERSION/tutorials/tutorial-ingestion-spec.html) -* [Tutorial: Transforming input data](/docs/VERSION/tutorials/tutorial-transform-spec.html) +* [Tutorial: Transforming input data](/docs/VERSION/tutorials/tutorial-transform-spec.html) * [Clustering](/docs/VERSION/tutorials/cluster.html) +* Further examples + * [Single-server deployment](/docs/VERSION/operations/single-server.html) + * [Clustered deployment](/docs/VERSION/operations/example-cluster.html) ## Data Ingestion * [Ingestion overview](/docs/VERSION/ingestion/index.html) @@ -104,14 +107,17 @@ layout: toc * [Overview](/docs/VERSION/design/index.html) * Storage * [Segments](/docs/VERSION/design/segments.html) - * [Processes and Servers](/docs/VERSION/design/processes.html) -* [Coordinator](/docs/VERSION/design/coordinator.html) -* [Overlord](/docs/VERSION/design/overlord.html) -* [Broker](/docs/VERSION/design/broker.html) -* [Historical](/docs/VERSION/design/historical.html) -* [MiddleManager](/docs/VERSION/design/middlemanager.html) - * [Peons](/docs/VERSION/design/peons.html) -* [Router](/docs/VERSION/development/router.html) (optional; experimental) + * [Servers and Processes](/docs/VERSION/design/processes.html) +* Master server + * [Coordinator](/docs/VERSION/design/coordinator.html) + * [Overlord](/docs/VERSION/design/overlord.html) +* Query server + * [Broker](/docs/VERSION/design/broker.html) + * [Router](/docs/VERSION/development/router.html) (optional; experimental) +* Data server + * [Historical](/docs/VERSION/design/historical.html) + * [MiddleManager](/docs/VERSION/design/middlemanager.html) +* [Peons](/docs/VERSION/design/peons.html) * Dependencies * [Deep Storage](/docs/VERSION/dependencies/deep-storage.html) * [Metadata Storage](/docs/VERSION/dependencies/metadata-storage.html) @@ -126,20 +132,19 @@ layout: toc * [Metrics and Monitoring](/docs/VERSION/operations/metrics.html) * [Alerts](/docs/VERSION/operations/alerts.html) * [Different Hadoop Versions](/docs/VERSION/operations/other-hadoop.html) - * [HTTP Compression](/docs/VERSION/operations/http-compression.html) - * [Basic Cluster Tuning](/docs/VERSION/operations/basic-cluster-tuning.html) - * Examples - * [Single-server Deployment Examples](/docs/VERSION/operations/single-server.html) - * [Clustered Deployment Example](/docs/VERSION/operations/example-cluster.html) - * [Recommendations](/docs/VERSION/operations/recommendations.html) -* [Performance
[GitHub] [incubator-druid] fjy commented on issue #7628: 0.14.2-incubating release notes
fjy commented on issue #7628: 0.14.2-incubating release notes URL: https://github.com/apache/incubator-druid/issues/7628#issuecomment-495314956 We still need 2 more +1 votes: https://lists.apache.org/list.html?gene...@incubator.apache.org This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] AlexanderSaydakov commented on issue #7628: 0.14.2-incubating release notes
AlexanderSaydakov commented on issue #7628: 0.14.2-incubating release notes URL: https://github.com/apache/incubator-druid/issues/7628#issuecomment-495314738 What is stopping this release? It has been two weeks since release candidate. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on a change in pull request #7138: 'core' ORC extension
leventov commented on a change in pull request #7138: 'core' ORC extension
URL: https://github.com/apache/incubator-druid/pull/7138#discussion_r287044764
##
File path:
extensions-core/orc-extensions/src/main/java/org/apache/druid/data/input/orc/OrcStructConverter.java
##
@@ -0,0 +1,239 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input.orc;
+
+import it.unimi.dsi.fastutil.objects.Object2IntMap;
+import it.unimi.dsi.fastutil.objects.Object2IntOpenHashMap;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.hadoop.hive.serde2.io.DateWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.io.BooleanWritable;
+import org.apache.hadoop.io.ByteWritable;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.io.DoubleWritable;
+import org.apache.hadoop.io.FloatWritable;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.io.ShortWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.io.WritableComparable;
+import org.apache.orc.TypeDescription;
+import org.apache.orc.mapred.OrcList;
+import org.apache.orc.mapred.OrcMap;
+import org.apache.orc.mapred.OrcStruct;
+import org.apache.orc.mapred.OrcTimestamp;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+public class OrcStructConverter
+{
+ @Nonnull
+ private static List convertList(TypeDescription fieldDescription,
OrcList orcList, boolean binaryAsString)
+ {
+// if primitive list, convert primitives
+TypeDescription listType = fieldDescription.getChildren().get(0);
+if (listType.getCategory().isPrimitive()) {
+ return (List) orcList.stream()
+ .map(li -> convertPrimitive(listType,
(WritableComparable) li, binaryAsString))
+ .collect(Collectors.toList());
+}
+return new ArrayList(orcList);
+ }
+
+ private static Map convertMap(
+ TypeDescription fieldDescription,
+ OrcMap map,
+ boolean binaryAsString
+ )
+ {
+Map converted = new HashMap<>();
+TypeDescription keyDescription = fieldDescription.getChildren().get(0);
+TypeDescription valueDescription = fieldDescription.getChildren().get(1);
+for (WritableComparable key : map.navigableKeySet()) {
+ Object newKey = convertPrimitive(keyDescription, key, binaryAsString);
+ if (valueDescription.getCategory().isPrimitive()) {
+converted.put(newKey, convertPrimitive(valueDescription, map.get(key),
binaryAsString));
+ } else {
+converted.put(newKey, map.get(key));
+ }
+}
+return converted;
+ }
+
+ private static Object convertPrimitive(TypeDescription fieldDescription,
WritableComparable field, boolean binaryAsString)
+ {
+/*
+ORC TYPEWRITABLE TYPE
+binary org.apache.hadoop.io.BytesWritable
+bigint org.apache.hadoop.io.LongWritable
+boolean org.apache.hadoop.io.BooleanWritable
+charorg.apache.hadoop.io.Text
+dateorg.apache.hadoop.hive.serde2.io.DateWritable
+decimal org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
+double org.apache.hadoop.io.DoubleWritable
+float org.apache.hadoop.io.FloatWritable
+int org.apache.hadoop.io.IntWritable
+smallintorg.apache.hadoop.io.ShortWritable
+string org.apache.hadoop.io.Text
+timestamp org.apache.orc.mapred.OrcTimestamp
+tinyint org.apache.hadoop.io.ByteWritable
+varchar org.apache.hadoop.io.Text
+ */
+switch (fieldDescription.getCategory()) {
+ case STRING:
+ case CHAR:
+ case VARCHAR:
+return ((Text) field).toString();
+ case BOOLEAN:
+return ((BooleanWritable) field).get();
+ case BYTE:
+return ((ByteWritable) field).get();
+ case SHORT:
+return
[GitHub] [incubator-druid] jihoonson commented on issue #7734: reorganizing the ToC
jihoonson commented on issue #7734: reorganizing the ToC URL: https://github.com/apache/incubator-druid/pull/7734#issuecomment-495302516 Sure, raised https://github.com/apache/incubator-druid/pull/7739. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson opened a new pull request #7739: [Backport] reorganizing the ToC
jihoonson opened a new pull request #7739: [Backport] reorganizing the ToC URL: https://github.com/apache/incubator-druid/pull/7739 Backport of #7734 to 0.15.0-incubating. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #7609: Local variable names shouldn't start with capital
leventov commented on issue #7609: Local variable names shouldn't start with capital URL: https://github.com/apache/incubator-druid/issues/7609#issuecomment-495295982 @sashidhar it looks like there is a collision and @lml2468 has already solved this problem. In general, please consult to Github Help and web search to resolve such questions: https://help.github.com/en/articles/creating-a-pull-request This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #7681: Add checkstyle for "Local variable names shouldn't start with capital"
leventov commented on issue #7681: Add checkstyle for "Local variable names shouldn't start with capital" URL: https://github.com/apache/incubator-druid/pull/7681#issuecomment-495294723 @lml2468 thanks for contribution! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov closed issue #7609: Local variable names shouldn't start with capital
leventov closed issue #7609: Local variable names shouldn't start with capital URL: https://github.com/apache/incubator-druid/issues/7609 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
