[druid] branch master updated (a1f850d -> 3b9dad4)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from a1f850d  Fix vectorized cardinality bug on certain string columns. 
(#11199)
 add 3b9dad4  Consolidate the number of Dockerfiles (#11187)

No new revisions were added by this update.

Summary of changes:
 distribution/docker/Dockerfile | 16 ---
 distribution/docker/Dockerfile.java11  | 61 --
 distribution/docker/DockerfileBuildTarAdvanced | 43 ++
 integration-tests/script/setup_druid_on_k8s.sh |  4 +-
 4 files changed, 16 insertions(+), 108 deletions(-)
 delete mode 100644 distribution/docker/Dockerfile.java11

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated: K8s IT Test enhance (#10785)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 77946f9  K8s IT Test enhance  (#10785)
77946f9 is described below

commit 77946f92649c8deba307f99eaf5320d6e76fe390
Author: zhangyue19921010 <69956021+zhangyue19921...@users.noreply.github.com>
AuthorDate: Tue Feb 2 07:48:42 2021 +0800

K8s IT Test enhance  (#10785)

* do build and stop action in IT

* change base dir from druidHome to druidHome/integration-tests

* add env DRUID_HOME

* bug fix

* modify stop_sh

* ready to test

* bug fix

* modify dir

* tested on dev

* modify dir

* move DRUID_HOME env

* done

Co-authored-by: yuezhang 
---
 .travis.yml|  9 +
 distribution/docker/DockerfileBuildTarAdvanced |  4 ++
 integration-tests/pom.xml  | 46 ++
 ...top_k8s_cluster.sh => build_run_k8s_cluster.sh} | 25 +++-
 integration-tests/script/setup_druid_on_k8s.sh |  3 +-
 .../script/setup_druid_operator_on_k8s.sh  |  1 +
 integration-tests/script/stop_k8s_cluster.sh   | 15 ++-
 7 files changed, 91 insertions(+), 12 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index d5c9391..ffb6c1d 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -601,14 +601,9 @@ jobs:
   jdk: openjdk8
   services: _test_services_k8s
 - docker
-  env: CONFIG_FILE='k8s_run_config_file.json' 
IT_TEST='-Dit.test=ITNestedQueryPushDownTest' POD_NAME=int-test 
POD_NAMESPACE=default
-  before_script:
-- integration-tests/script/setup_k8s_cluster.sh
-- integration-tests/script/setup_druid_operator_on_k8s.sh
-- integration-tests/script/setup_druid_on_k8s.sh
+  env: CONFIG_FILE='k8s_run_config_file.json' 
IT_TEST='-Dit.test=ITNestedQueryPushDownTest' POD_NAME=int-test 
POD_NAMESPACE=default BUILD_DRUID_CLSUTER=true
   script: _integration_test_k8s
-- ${MVN} verify -pl integration-tests -P int-tests-config-file 
${IT_TEST} ${MAVEN_SKIP}
-  after_script: integration-tests/script/stop_k8s_cluster.sh
+- ${MVN} verify -pl integration-tests -P int-tests-config-file 
${IT_TEST} ${MAVEN_SKIP} -Dpod.name=${POD_NAME} 
-Dpod.namespace=${POD_NAMESPACE} -Dbuild.druid.cluster=${BUILD_DRUID_CLSUTER}
   after_failure: _test_diags_k8s
 - for v in broker middlemanager router coordinator historical ; do
   echo 
"druid-tiny-cluster-"$v"s-0-";
diff --git a/distribution/docker/DockerfileBuildTarAdvanced 
b/distribution/docker/DockerfileBuildTarAdvanced
index a58087f..0c04cd2 100644
--- a/distribution/docker/DockerfileBuildTarAdvanced
+++ b/distribution/docker/DockerfileBuildTarAdvanced
@@ -55,6 +55,10 @@ RUN addgroup -S -g 1000 druid \
 COPY --chown=druid:druid --from=builder /opt /opt
 COPY distribution/docker/druid.sh /druid.sh
 
+RUN mkdir -p /opt/druid/var \
+ && chown -R druid:druid /opt \
+ && chmod 775 /opt/druid/var
+
 USER druid
 VOLUME /opt/druid/var
 WORKDIR /opt/druid
diff --git a/integration-tests/pom.xml b/integration-tests/pom.xml
index ddcdaaa..b47bb9c 100644
--- a/integration-tests/pom.xml
+++ b/integration-tests/pom.xml
@@ -498,9 +498,55 @@
 
 
 int-tests-config-file
+
+int-test
+default
+false
+
+
+
+
+\ %Россия\ 한국\ 
中国!?
+
 
 
 
+org.codehaus.mojo
+exec-maven-plugin
+
+
+K8s-package
+
+exec
+
+pre-integration-test
+
+
+${pod.name}
+
${pod.namespace}
+
${build.druid.cluster}
+
+
${project.basedir}/script/build_run_k8s_cluster.sh
+
+
+
+stop-k8s-druid-cluster
+
+exec
+
+post-integration-test
+
+
+${pod.name}
+   

[druid] branch master updated (48e576a -> c7b1212)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 48e576a  Scan query: More accurate error message when segment per time 
chunk limit is exceeded. (#10630)
 add c7b1212  AWS RDS token based password  provider (#9518)

No new revisions were added by this update.

Summary of changes:
 distribution/pom.xml   |   2 +
 docs/development/extensions-core/druid-aws-rds.md  |  38 +
 docs/development/extensions.md |   1 +
 .../pom.xml|  45 --
 .../org/apache/druid/aws/rds/AWSRDSModule.java}|  11 +-
 .../druid/aws/rds/AWSRDSTokenPasswordProvider.java | 123 ++
 .../org.apache.druid.initialization.DruidModule|   3 +-
 .../aws/rds/AWSRDSTokenPasswordProviderTest.java   |  82 ++
 licenses.yaml  |  10 ++
 pom.xml|   4 +-
 server/pom.xml |  11 ++
 .../apache/druid/metadata/BasicDataSourceExt.java  | 179 +
 .../metadata/SQLFirehoseDatabaseConnector.java |   2 +-
 .../druid/metadata/SQLMetadataConnector.java   |   2 +-
 .../druid/metadata/BasicDataSourceExtTest.java | 113 +
 website/.spelling  |   3 +
 16 files changed, 601 insertions(+), 28 deletions(-)
 create mode 100644 docs/development/extensions-core/druid-aws-rds.md
 copy extensions-core/{simple-client-sslcontext => 
druid-aws-rds-extensions}/pom.xml (75%)
 copy 
extensions-core/{s3-extensions/src/main/java/org/apache/druid/firehose/s3/S3FirehoseDruidModule.java
 => 
druid-aws-rds-extensions/src/main/java/org/apache/druid/aws/rds/AWSRDSModule.java}
 (84%)
 create mode 100644 
extensions-core/druid-aws-rds-extensions/src/main/java/org/apache/druid/aws/rds/AWSRDSTokenPasswordProvider.java
 copy services/src/bin/run.sh => 
extensions-core/druid-aws-rds-extensions/src/main/resources/META-INF/services/org.apache.druid.initialization.DruidModule
 (95%)
 create mode 100644 
extensions-core/druid-aws-rds-extensions/src/test/java/org/apache/druid/aws/rds/AWSRDSTokenPasswordProviderTest.java
 create mode 100644 
server/src/main/java/org/apache/druid/metadata/BasicDataSourceExt.java
 create mode 100644 
server/src/test/java/org/apache/druid/metadata/BasicDataSourceExtTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (ea2d51d -> d2e6240)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from ea2d51d  Better error message for compaction task when it sees no 
segments for compaction (#10728)
 add d2e6240  k8s-int-test-build: zk-less druid cluster and http based 
segment/task managment (#10686)

No new revisions were added by this update.

Summary of changes:
 .travis.yml|  7 +-
 docs/development/extensions-core/kubernetes.md | 35 +++-
 .../k8s/role-and-binding.yaml  | 31 +--
 integration-tests/{ => k8s}/tiny-cluster.yaml  | 49 ---
 integration-tests/script/build_run_k8s_cluster.sh  | 84 ---
 integration-tests/script/setup_druid_on_k8s.sh | 52 
 .../setup_druid_operator_on_k8s.sh}| 34 
 .../setup_k8s_cluster.sh}  | 33 +++-
 integration-tests/tiny-cluster-zk.yaml | 97 --
 9 files changed, 179 insertions(+), 243 deletions(-)
 copy .lgtm.yml => integration-tests/k8s/role-and-binding.yaml (66%)
 rename integration-tests/{ => k8s}/tiny-cluster.yaml (91%)
 delete mode 100755 integration-tests/script/build_run_k8s_cluster.sh
 create mode 100755 integration-tests/script/setup_druid_on_k8s.sh
 copy integration-tests/{build_run_cluster.sh => 
script/setup_druid_operator_on_k8s.sh} (50%)
 copy integration-tests/{build_run_cluster.sh => script/setup_k8s_cluster.sh} 
(51%)
 delete mode 100644 integration-tests/tiny-cluster-zk.yaml


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (55b8cc4 -> 1884c35)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 55b8cc4  remove extra word (#10682)
 add 1884c35  Do Integrate test for Druid base on K8s cluster (#10669)

No new revisions were added by this update.

Summary of changes:
 .travis.yml|  15 +
 .../{Dockerfile => DockerfileBuildTarAdvanced} |  11 +-
 integration-tests/k8s_run_config_file.json |  16 ++
 integration-tests/script/build_run_k8s_cluster.sh  |  84 ++
 .../script/stop_k8s_cluster.sh |   5 +-
 integration-tests/tiny-cluster-zk.yaml |  97 +++
 integration-tests/tiny-cluster.yaml| 320 +
 7 files changed, 542 insertions(+), 6 deletions(-)
 copy distribution/docker/{Dockerfile => DockerfileBuildTarAdvanced} (86%)
 create mode 100644 integration-tests/k8s_run_config_file.json
 create mode 100755 integration-tests/script/build_run_k8s_cluster.sh
 copy 
extensions-contrib/ambari-metrics-emitter/src/main/resources/META-INF/services/org.apache.druid.initialization.DruidModule
 => integration-tests/script/stop_k8s_cluster.sh (90%)
 mode change 100644 => 100755
 create mode 100644 integration-tests/tiny-cluster-zk.yaml
 create mode 100644 integration-tests/tiny-cluster.yaml


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (c2e26d2 -> ac1882b)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from c2e26d2  Add status/selfDiscovered endpoint to indexer for self 
discovery of indexer (#10679)
 add ac1882b  kubernetes based discovery druid extension to run Druid on 
K8S without Zookeeper (#10544)

No new revisions were added by this update.

Summary of changes:
 .../druid/annotations/SuppressFBWarnings.java  |  42 +++
 distribution/bin/check-licenses.py |   1 +
 distribution/pom.xml   |   2 +
 docs/development/extensions-core/kubernetes.md |  59 
 docs/operations/kubernetes.md  |   2 +
 extensions-core/kubernetes-extensions/pom.xml  | 152 +
 .../druid/k8s/discovery/DefaultK8sApiClient.java   | 186 +++
 .../discovery/DefaultK8sLeaderElectorFactory.java  |  91 ++
 .../DiscoveryDruidNodeAndResourceVersion.java  |  44 +++
 .../k8s/discovery/DiscoveryDruidNodeList.java  |  52 +++
 .../apache/druid/k8s/discovery/K8sApiClient.java   |  37 +++
 .../druid/k8s/discovery/K8sDiscoveryConfig.java| 205 
 .../druid/k8s/discovery/K8sDiscoveryModule.java| 152 +
 .../k8s/discovery/K8sDruidLeaderSelector.java  | 152 +
 .../druid/k8s/discovery/K8sDruidNodeAnnouncer.java | 266 +++
 .../discovery/K8sDruidNodeDiscoveryProvider.java   | 363 +
 .../k8s/discovery/K8sLeaderElectorFactory.java |  41 +++
 .../k8s/discovery/LeaderElectorAsyncWrapper.java   | 119 +++
 .../org/apache/druid/k8s/discovery/PodInfo.java|  59 
 .../apache/druid/k8s/discovery/WatchResult.java|  36 ++
 .../org.apache.druid.initialization.DruidModule|  16 +
 .../discovery/K8sAnnouncerAndDiscoveryIntTest.java | 124 +++
 .../k8s/discovery/K8sDiscoveryConfigTest.java  |  79 +
 .../discovery/K8sDruidLeaderElectionIntTest.java   | 166 ++
 .../k8s/discovery/K8sDruidLeaderSelectorTest.java  | 180 ++
 .../k8s/discovery/K8sDruidNodeAnnouncerTest.java   | 143 
 .../K8sDruidNodeDiscoveryProviderTest.java | 343 +++
 .../druid/k8s/discovery/NoopServiceEmitter.java|  36 ++
 licenses.yaml  | 250 ++
 pom.xml|   1 +
 .../CuratorDruidNodeDiscoveryProvider.java | 184 +--
 .../druid/discovery/BaseNodeRoleWatcher.java   | 301 +
 .../druid/discovery/DruidLeaderSelector.java   |   1 +
 .../druid/discovery/BaseNodeRoleWatcherTest.java   | 161 +
 .../main/java/org/apache/druid/cli/CliBroker.java  |  15 +-
 .../main/java/org/apache/druid/cli/CliIndexer.java |   5 +-
 .../main/java/org/apache/druid/cli/CliPeon.java|  15 +-
 website/.spelling  |   8 +
 38 files changed, 3910 insertions(+), 179 deletions(-)
 create mode 100644 
core/src/main/java/org/apache/druid/annotations/SuppressFBWarnings.java
 create mode 100644 docs/development/extensions-core/kubernetes.md
 create mode 100644 extensions-core/kubernetes-extensions/pom.xml
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sLeaderElectorFactory.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DiscoveryDruidNodeAndResourceVersion.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DiscoveryDruidNodeList.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sApiClient.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDiscoveryConfig.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDiscoveryModule.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidLeaderSelector.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeDiscoveryProvider.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sLeaderElectorFactory.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/LeaderElectorAsyncWrapper.java
 create mode 100644 
extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/PodInfo.java
 create mode 100644 
extensions-core/kubernetes-extensions

[druid] branch master updated (64f97e7 -> 0ad27c0)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 64f97e7  fix DruidSchema incorrectly listing tables with no segments 
(#10660)
 add 0ad27c0  Historical load Segments enhancement (#10650)

No new revisions were added by this update.

Summary of changes:
 .../loading/SegmentLoaderLocalCacheManager.java| 36 +-
 .../coordination/SegmentLoadDropHandler.java   |  5 ++-
 2 files changed, 39 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated: document DynamicConfigProvider for kafka consumer properties (#10658)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new be01976  document DynamicConfigProvider for kafka consumer properties 
(#10658)
be01976 is described below

commit be019760bb878e33c4bce49d9d724b7fd789be4b
Author: Himanshu 
AuthorDate: Thu Dec 10 08:24:33 2020 -0800

document DynamicConfigProvider for kafka consumer properties (#10658)

* document DynamicConfigProvider for kafka consumer properties

* Update docs/development/extensions-core/kafka-ingestion.md

Co-authored-by: Jihoon Son 

* Update docs/development/extensions-core/kafka-ingestion.md

* fix doc build

Co-authored-by: Jihoon Son 
---
 .../development/extensions-core/kafka-ingestion.md | 14 -
 docs/development/modules.md| 18 
 docs/operations/dynamic-config-provider.md | 33 ++
 website/.spelling  |  2 ++
 4 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/docs/development/extensions-core/kafka-ingestion.md 
b/docs/development/extensions-core/kafka-ingestion.md
index c128c1b..edf3b6e 100644
--- a/docs/development/extensions-core/kafka-ingestion.md
+++ b/docs/development/extensions-core/kafka-ingestion.md
@@ -134,7 +134,7 @@ A sample supervisor spec is shown below:
 |-||---||
 |`topic`|String|The Kafka topic to read from. This must be a specific topic as 
topic patterns are not supported.|yes|
 
|`inputFormat`|Object|[`inputFormat`](../../ingestion/data-formats.md#input-format)
 to specify how to parse input data. See [the below 
section](#specifying-data-format) for details about specifying the input 
format.|yes|
-|`consumerProperties`|Map|A map of properties to be passed to 
the Kafka consumer. This must contain a property `bootstrap.servers` with a 
list of Kafka brokers in the form: 
`:,:,...`. Users can set `isolation.level` 
`read_uncommitted` here if don't need Druid to consume transactional topics or 
need Druid to consume older versions of Kafka. For SSL connections, the 
`keystore`, `truststore` and `key` passwords can be provided as a [Passwo [...]
+|`consumerProperties`|Map|A map of properties to be passed to 
the Kafka consumer. See [next section](#more-on-consumerproperties) for more 
information.|yes|
 |`pollTimeout`|Long|The length of time to wait for the Kafka consumer to poll 
records, in milliseconds|no (default == 100)|
 |`replicas`|Integer|The number of replica sets, where 1 means a single set of 
tasks (no replication). Replica tasks will always be assigned to different 
workers to provide resiliency against process failure.|no (default == 1)|
 |`taskCount`|Integer|The maximum number of *reading* tasks in a *replica set*. 
This means that the maximum number of reading tasks will be `taskCount * 
replicas` and the total number of tasks (*reading* + *publishing*) will be 
higher than this. See [Capacity Planning](#capacity-planning) below for more 
details. The number of reading tasks will be less than `taskCount` if 
`taskCount > {numKafkaPartitions}`.|no (default == 1)|
@@ -147,6 +147,18 @@ A sample supervisor spec is shown below:
 |`lateMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject 
messages with timestamps earlier than this period before the task was created; 
for example if this is set to `PT1H` and the supervisor creates a task at 
*2016-01-01T12:00Z*, messages with timestamps earlier than *2016-01-01T11:00Z* 
will be dropped. This may help prevent concurrency issues if your data stream 
has late messages and you have multiple pipelines that need to operate on the 
same segments (e.g. a realtime an [...]
 |`earlyMessageRejectionPeriod`|ISO8601 Period|Configure tasks to reject 
messages with timestamps later than this period after the task reached its 
taskDuration; for example if this is set to `PT1H`, the taskDuration is set to 
`PT1H` and the supervisor creates a task at *2016-01-01T12:00Z*, messages with 
timestamps later than *2016-01-01T14:00Z* will be dropped. **Note:** Tasks 
sometimes run past their task duration, for example, in cases of supervisor 
failover. Setting earlyMessageReject [...]
 
+ More on consumerProperties
+
+This must contain a property `bootstrap.servers` with a list of Kafka brokers 
in the form: `:,:,...`.
+By default, `isolation.level` is set to `read_committed`. It should be set to 
`read_uncommitted` if you don't want Druid to consume only committed 
transactions or working with older versions of Kafka servers with no 
Transactions support.
+
+There are few cases that require fetching few/all of consumer properties at 
runtime e.g. when `bootstrap.servers` is not known upfront or not static, to 
enable SSL connections users might have to provide passwords for `keyst

[druid] branch master updated (ae6c43d -> 229b5f3)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from ae6c43d  Add an integration test for HTTP inputSource (#10620)
 add 229b5f3  Remove hard limitation that druid(after 0.15.0) only can 
consume Kafka version 0.11.x or better (#10551)

No new revisions were added by this update.

Summary of changes:
 .../development/extensions-core/kafka-ingestion.md |  6 ++--
 .../druid/indexing/kafka/KafkaConsumerConfigs.java |  1 -
 .../druid/indexing/kafka/KafkaIndexTask.java   |  1 +
 .../druid/indexing/kafka/KafkaRecordSupplier.java  |  1 +
 .../druid/indexing/kafka/KafkaIndexTaskTest.java   | 41 ++
 5 files changed, 47 insertions(+), 3 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (c94be8a -> 813e187)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from c94be8a  Revert "Update google client libraries (#10536)" (#10599)
 add 813e187  make dimension column extensible with COMPLEX type (#10277)

No new revisions were added by this update.

Summary of changes:
 ...tSegment.java => DimensionHandlerProvider.java} |   9 +-
 .../druid/segment/DimensionHandlerUtils.java   |  31 ++
 .../apache/druid/segment/column/ColumnBuilder.java |   6 ++
 .../druid/segment/column/ColumnCapabilities.java   |   7 ++
 .../segment/column/ColumnCapabilitiesImpl.java |  25 +
 .../segment/incremental/IncrementalIndex.java  |  13 ++-
 .../segment/serde/ComplexColumnPartSerde.java  |   2 +-
 .../druid/segment/serde/ComplexMetricSerde.java|  16 
 .../query/aggregation/AggregationTestHelper.java   | 106 ++---
 ...ilsTest.java => DimensionHandlerUtilsTest.java} |  36 ---
 .../java/org/apache/druid/segment/TestIndex.java   |   6 +-
 .../druid/segment/column/ColumnBuilderTest.java|  16 ++--
 ...ColumnSupportedComplexColumnSerializerTest.java |   2 +-
 13 files changed, 219 insertions(+), 56 deletions(-)
 copy processing/src/main/java/org/apache/druid/segment/{AbstractSegment.java 
=> DimensionHandlerProvider.java} (76%)
 copy processing/src/test/java/org/apache/druid/segment/{IntListUtilsTest.java 
=> DimensionHandlerUtilsTest.java} (56%)
 copy 
extensions-contrib/ambari-metrics-emitter/src/test/java/org/apache/druid/emitter/ambari/metrics/DruidToWhiteListBasedConverterTest.java
 => 
processing/src/test/java/org/apache/druid/segment/column/ColumnBuilderTest.java 
(72%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (f965464 -> 7e95228)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from f965464  Fix empty directory handling (#10319)
 add 7e95228  introduce DynamicConfigProvider interface and make kafka 
consumer props extensible (#10309)

No new revisions were added by this update.

Summary of changes:
 ...ordProvider.java => DynamicConfigProvider.java} | 13 
 .../metadata/MapStringDynamicConfigProvider.java   | 28 -
 .../apache/druid/metadata/PasswordProvider.java|  6 
 .../MapStringDynamicConfigProviderTest.java| 24 +++
 .../druid/indexing/kafka/KafkaRecordSupplier.java  | 35 +++---
 .../kafka/supervisor/KafkaSupervisorIOConfig.java  |  1 +
 .../indexing/kafka/KafkaRecordSupplierTest.java| 31 +++
 7 files changed, 94 insertions(+), 44 deletions(-)
 copy core/src/main/java/org/apache/druid/metadata/{PasswordProvider.java => 
DynamicConfigProvider.java} (76%)
 copy 
extensions-core/hdfs-storage/src/main/java/org/apache/druid/storage/hdfs/tasklog/HdfsTaskLogsConfig.java
 => 
core/src/main/java/org/apache/druid/metadata/MapStringDynamicConfigProvider.java
 (67%)
 copy 
server/src/test/java/org/apache/druid/discovery/LookupNodeServiceTest.java => 
core/src/test/java/org/apache/druid/metadata/MapStringDynamicConfigProviderTest.java
 (62%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated: Add new coordinator metrics for coordinator duty runtimes (#10603)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 2560bf0  Add new coordinator metrics for coordinator duty runtimes 
(#10603)
2560bf0 is described below

commit 2560bf0a1919c36b824bd0e4f9286e2899deddd3
Author: Lucas Capistrant 
AuthorDate: Sun Nov 29 16:47:35 2020 -0600

Add new coordinator metrics for coordinator duty runtimes (#10603)

* Add new coordinator metrics for duty runtimes

* fix spelling for a constant variable value

* add comment clarifying why the global runtime metric is emitted where it 
is

* Remove duty alias in lieu of using the class name for metrics

* fix docs

* CoordinatorStats tests + add duty stats to accumulate() logic
---
 docs/operations/metrics.md |  2 +
 .../java/org/apache/druid/query/DruidMetrics.java  |  3 ++
 .../druid/server/coordinator/CoordinatorStats.java | 51 ++
 .../druid/server/coordinator/DruidCoordinator.java | 28 --
 .../duty/EmitClusterStatsAndMetrics.java   | 32 +++
 .../server/coordinator/CoordinatorStatsTest.java   | 63 ++
 .../server/coordinator/DruidCoordinatorTest.java   |  2 +-
 7 files changed, 176 insertions(+), 5 deletions(-)

diff --git a/docs/operations/metrics.md b/docs/operations/metrics.md
index 62c0521..68b7f88 100644
--- a/docs/operations/metrics.md
+++ b/docs/operations/metrics.md
@@ -254,6 +254,8 @@ These metrics are for the Druid Coordinator and are reset 
each time the Coordina
 |`segment/skipCompact/bytes`|Total bytes of this datasource that are skipped 
(not eligible for auto compaction) by the auto compaction.|datasource.|Varies.|
 |`segment/skipCompact/count`|Total number of segments of this datasource that 
are skipped (not eligible for auto compaction) by the auto 
compaction.|datasource.|Varies.|
 |`interval/skipCompact/count`|Total number of intervals of this datasource 
that are skipped (not eligible for auto compaction) by the auto 
compaction.|datasource.|Varies.|
+|`coordinator/time`|Approximate Coordinator duty runtime in milliseconds. The 
duty dimension is the string alias of the Duty that is being run.|duty.|Varies.|
+|`coordinator/global/time`|Approximate runtime of a full coordination cycle in 
milliseconds. The `dutyGroup` dimension indicates what type of coordination 
this run was. i.e. Historical Management vs Indexing|`dutyGroup`|Varies.|
 
 If `emitBalancingStats` is set to `true` in the Coordinator [dynamic 
configuration](
 ../configuration/index.html#dynamic-configuration), then [log 
entries](../configuration/logging.md) for class
diff --git a/processing/src/main/java/org/apache/druid/query/DruidMetrics.java 
b/processing/src/main/java/org/apache/druid/query/DruidMetrics.java
index b1e49af..1e31c5f 100644
--- a/processing/src/main/java/org/apache/druid/query/DruidMetrics.java
+++ b/processing/src/main/java/org/apache/druid/query/DruidMetrics.java
@@ -42,6 +42,9 @@ public class DruidMetrics
   public static final String SERVER = "server";
   public static final String TIER = "tier";
 
+  public static final String DUTY = "duty";
+  public static final String DUTY_GROUP = "dutyGroup";
+
   public static int findNumComplexAggs(List aggs)
   {
 int retVal = 0;
diff --git 
a/server/src/main/java/org/apache/druid/server/coordinator/CoordinatorStats.java
 
b/server/src/main/java/org/apache/druid/server/coordinator/CoordinatorStats.java
index fa10f20..357cf45 100644
--- 
a/server/src/main/java/org/apache/druid/server/coordinator/CoordinatorStats.java
+++ 
b/server/src/main/java/org/apache/druid/server/coordinator/CoordinatorStats.java
@@ -35,12 +35,14 @@ public class CoordinatorStats
 {
   private final Map> perTierStats;
   private final Map> perDataSourceStats;
+  private final Map> perDutyStats;
   private final Object2LongOpenHashMap globalStats;
 
   public CoordinatorStats()
   {
 perTierStats = new HashMap<>();
 perDataSourceStats = new HashMap<>();
+perDutyStats = new HashMap<>();
 globalStats = new Object2LongOpenHashMap<>();
   }
 
@@ -54,6 +56,11 @@ public class CoordinatorStats
 return !perDataSourceStats.isEmpty();
   }
 
+  public boolean hasPerDutyStats()
+  {
+return !perDutyStats.isEmpty();
+  }
+
   public Set getTiers(final String statName)
   {
 final Object2LongOpenHashMap theStat = perTierStats.get(statName);
@@ -72,6 +79,15 @@ public class CoordinatorStats
 return Collections.unmodifiableSet(stat.keySet());
   }
 
+  public Set getDuties(String statName)
+  {
+final Object2LongOpenHashMap stat = perDutyStats.get(statName);
+if (stat == null) {
+  return Collections.emptySet();
+}
+return Collections.unmodifiableSet(stat.keySet());
+  }

[druid] branch master updated: DataSourcesSnapshotBenchmark to measure iterateAllUsedSegmentsInSnapshot perf (#10604)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 30bcb0f  DataSourcesSnapshotBenchmark to measure 
iterateAllUsedSegmentsInSnapshot perf (#10604)
30bcb0f is described below

commit 30bcb0fd74daf4f50f7e81d8acaea9577b03cbb3
Author: Himanshu 
AuthorDate: Sun Nov 29 14:42:14 2020 -0800

DataSourcesSnapshotBenchmark to measure iterateAllUsedSegmentsInSnapshot 
perf (#10604)
---
 .../benchmark/DataSourcesSnapshotBenchmark.java| 124 +
 1 file changed, 124 insertions(+)

diff --git 
a/benchmarks/src/test/java/org/apache/druid/benchmark/DataSourcesSnapshotBenchmark.java
 
b/benchmarks/src/test/java/org/apache/druid/benchmark/DataSourcesSnapshotBenchmark.java
new file mode 100644
index 000..ab6fff0
--- /dev/null
+++ 
b/benchmarks/src/test/java/org/apache/druid/benchmark/DataSourcesSnapshotBenchmark.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark;
+
+import org.apache.druid.client.DataSourcesSnapshot;
+import org.apache.druid.client.ImmutableDruidDataSource;
+import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.NoneShardSpec;
+import org.joda.time.Interval;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+@State(Scope.Benchmark)
+@Fork(1)
+@BenchmarkMode(Mode.SingleShotTime)
+@OutputTimeUnit(TimeUnit.NANOSECONDS)
+@Warmup(iterations = 10)
+@Measurement(iterations = 50)
+public class DataSourcesSnapshotBenchmark
+{
+  private static Interval TEST_SEGMENT_INTERVAL = 
Intervals.of("2012-03-15T00:00:00.000/2012-03-16T00:00:00.000");
+
+  @Param({"500", "1000"})
+  private int numDataSources;
+
+  @Param({"1000", "2000"})
+  private int numSegmentPerDataSource;
+
+  private DataSourcesSnapshot snapshot;
+
+  @Setup
+  public void setUp()
+  {
+long start = System.currentTimeMillis();
+
+Map dataSources = new HashMap<>();
+
+for (int i = 0; i < numDataSources; i++) {
+  String dataSource = StringUtils.format("ds-%d", i);
+  List segments = new ArrayList<>();
+
+  for (int j = 0; j < numSegmentPerDataSource; j++) {
+segments.add(
+new DataSegment(
+dataSource,
+TEST_SEGMENT_INTERVAL,
+String.valueOf(j),
+Collections.emptyMap(),
+Collections.emptyList(),
+Collections.emptyList(),
+NoneShardSpec.instance(),
+0,
+10L
+)
+);
+  }
+
+  dataSources.put(dataSource, new ImmutableDruidDataSource(dataSource, 
Collections.emptyMap(), segments));
+}
+
+snapshot = new DataSourcesSnapshot(dataSources);
+
+System.out.println("Setup Time " + (System.currentTimeMillis() - start) + 
" ms");
+  }
+
+  @Benchmark
+  public void iterateUsing_iterateAllUsedSegmentsInSnapshot(Blackhole 
blackhole)
+  {
+long totalSize = 0;
+for (DataSegment segment : snapshot.iterateAllUsedSegmentsInSnapshot()) {
+  totalSize += segment.getSize();
+}
+blackhole.consume(totalSize);
+  }
+
+  @Benchmark
+  public void 

[druid] branch master updated (31740b3 -> fe693a4)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 31740b3  Fix : Druid throws 
java.util.concurrent.RejectedExecutionException when ingest task is stopping. 
(#10555)
 add fe693a4  Improve doc and exception message for invalid user 
configurations (#10598)

No new revisions were added by this update.

Summary of changes:
 .../druid/java/util/common/HumanReadableBytes.java |  2 +-
 .../java/util/common/HumanReadableBytesRange.java  |  2 +-
 .../java/util/common/HumanReadableBytesTest.java   | 29 
 docs/configuration/index.md| 12 ++---
 .../overlord/config/RemoteTaskRunnerConfig.java|  7 ++-
 .../config/RemoteTaskRunnerConfigTest.java | 55 ++
 .../apache/druid/query/DruidProcessingConfig.java  | 14 --
 .../druid/query/DruidProcessingConfigTest.java | 42 +
 website/.spelling  |  3 ++
 9 files changed, 141 insertions(+), 25 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated: Fix : Druid throws java.util.concurrent.RejectedExecutionException when ingest task is stopping. (#10555)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 31740b3  Fix : Druid throws 
java.util.concurrent.RejectedExecutionException when ingest task is stopping. 
(#10555)
31740b3 is described below

commit 31740b3b29cac6d81d3cc19374dbf04836e41436
Author: zhangyue19921010 <69956021+zhangyue19921...@users.noreply.github.com>
AuthorDate: Tue Nov 24 06:52:03 2020 +0800

Fix : Druid throws java.util.concurrent.RejectedExecutionException when 
ingest task is stopping. (#10555)

* check exec status before return Signal

* add more log

* change log level to debug and add UT

* change log leverl to warn and merge master

Co-authored-by: yuezhang 
---
 .../util/common/concurrent/ScheduledExecutors.java |  7 +++-
 .../common/concurrent/ScheduledExecutorsTest.java  | 46 ++
 2 files changed, 52 insertions(+), 1 deletion(-)

diff --git 
a/core/src/main/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutors.java
 
b/core/src/main/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutors.java
index 2850c50..97f43f6 100644
--- 
a/core/src/main/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutors.java
+++ 
b/core/src/main/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutors.java
@@ -53,7 +53,12 @@ public class ScheduledExecutors
   public Signal call()
   {
 runnable.run(); // (Exceptions are handled for us)
-return Signal.REPEAT;
+if (exec.isShutdown()) {
+  log.warn("ScheduledExecutorService is ShutDown. Return 
'Signal.STOP' and stopped rescheduling %s (delay %s)", this, delay);
+  return Signal.STOP;
+} else {
+  return Signal.REPEAT;
+}
   }
 }
 );
diff --git 
a/core/src/test/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutorsTest.java
 
b/core/src/test/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutorsTest.java
new file mode 100644
index 000..9bc67ea
--- /dev/null
+++ 
b/core/src/test/java/org/apache/druid/java/util/common/concurrent/ScheduledExecutorsTest.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.concurrent;
+
+import org.joda.time.Duration;
+import org.junit.Test;
+
+import java.util.concurrent.ScheduledExecutorService;
+
+public class ScheduledExecutorsTest
+{
+  @Test
+  public void testscheduleWithFixedDelay() throws InterruptedException
+  {
+Duration initialDelay = new Duration(1000);
+Duration delay = new Duration(1000);
+ScheduledExecutorService exec = 
Execs.scheduledSingleThreaded("BasicAuthenticatorCacheManager-Exec--%d");
+ScheduledExecutors.scheduleWithFixedDelay(
+exec,
+initialDelay,
+delay,
+() -> {
+System.out.println("TEST!");
+}
+);
+Thread.sleep(5 * 1000);
+exec.shutdown();
+  }
+}


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (111b431 -> 4537016)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 111b431  Introduce query/timeout/count metric (#10567)
 add 4537016  Update google client libraries (#10536)

No new revisions were added by this update.

Summary of changes:
 licenses.yaml | 10 +-
 pom.xml   | 11 ++-
 2 files changed, 11 insertions(+), 10 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (3cafd53 -> 111b431)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 3cafd53  fix issue causing incorrect config in Docker (#10595)
 add 111b431  Introduce query/timeout/count metric (#10567)

No new revisions were added by this update.

Summary of changes:
 docs/operations/metrics.md |  9 +-
 .../src/main/resources/defaultMetrics.json |  1 +
 .../main/resources/defaultMetricDimensions.json|  1 +
 .../druid/server/AsyncQueryForwardingServlet.java  |  8 ++
 .../org/apache/druid/server/QueryResource.java |  9 +-
 .../server/metrics/QueryCountStatsMonitor.java |  7 +-
 .../server/metrics/QueryCountStatsProvider.java| 14 
 .../org/apache/druid/server/QueryResourceTest.java |  2 +
 .../server/metrics/QueryCountStatsMonitorTest.java | 96 ++
 9 files changed, 141 insertions(+), 6 deletions(-)
 create mode 100644 
server/src/test/java/org/apache/druid/server/metrics/QueryCountStatsMonitorTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (2201ffa -> 3cafd53)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 2201ffa  druid-docker-image: add DRUID_DIRS_TO_CREATE variable to 
customize directories created on startup (#10591)
 add 3cafd53  fix issue causing incorrect config in Docker (#10595)

No new revisions were added by this update.

Summary of changes:
 distribution/docker/druid.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (ba915b7 -> 2201ffa)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from ba915b7  Security overview documentation (#10339)
 add 2201ffa  druid-docker-image: add DRUID_DIRS_TO_CREATE variable to 
customize directories created on startup (#10591)

No new revisions were added by this update.

Summary of changes:
 distribution/docker/druid.sh | 7 ++-
 1 file changed, 6 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (aa9c0ec -> 4de4d4d)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from aa9c0ec  update quickstart docker-compose example, add to release 
instructions (#10527)
 add 4de4d4d  remove ServerDiscoverySelector from DruidLeaderClient (#10537)

No new revisions were added by this update.

Summary of changes:
 .../apache/druid/discovery/DruidLeaderClient.java   | 19 +--
 .../druid/guice/CoordinatorDiscoveryModule.java | 19 ++-
 .../druid/guice/IndexingServiceDiscoveryModule.java | 19 ++-
 .../druid/discovery/DruidLeaderClientTest.java  | 21 ++---
 .../coordinator/duty/CompactSegmentsTest.java   |  2 +-
 .../apache/druid/sql/calcite/util/CalciteTests.java |  5 +
 6 files changed, 13 insertions(+), 72 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (d0821de -> ee13630)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from d0821de  support for vectorizing expressions with non-existent inputs, 
more consistent type handling for non-vectorized expressions (#10499)
 add ee13630  optionally disable all of hardcoded zookeeper use (#9507)

No new revisions were added by this update.

Summary of changes:
 .../overlord/hrtr/HttpRemoteTaskRunner.java| 26 ---
 .../overlord/hrtr/HttpRemoteTaskRunnerFactory.java | 17 +--
 .../druid/indexing/worker/WorkerTaskManager.java   | 18 ++--
 .../worker/http/TaskManagementResource.java| 15 +++
 .../druid/indexing/worker/http/WorkerResource.java | 52 ++
 .../indexing/worker/http/WorkerResourceTest.java   |  6 ++-
 .../druid/cli/CliHistoricalForQueryRetryTest.java  |  1 +
 .../org/apache/druid/curator/CuratorModule.java|  7 ++-
 .../apache/druid/curator/ZkEnablementConfig.java   | 27 ++-
 .../druid/curator/discovery/DiscoveryModule.java   | 27 ---
 .../druid/curator/discovery/ServiceAnnouncer.java  | 16 +++
 .../org/apache/druid/guice/AnnouncerModule.java| 19 +++-
 .../coordination/BatchDataSegmentAnnouncer.java| 39 
 .../coordination/DataSegmentServerAnnouncer.java   | 16 +++
 .../druid/server/coordinator/DruidCoordinator.java | 28 
 .../server/coordinator/LoadQueueTaskMaster.java|  9 ++--
 .../druid/server/http/HistoricalResource.java  | 12 ++---
 .../coordinator/CuratorDruidCoordinatorTest.java   | 11 +++--
 .../server/coordinator/DruidCoordinatorTest.java   | 11 +++--
 .../java/org/apache/druid/cli/CliCoordinator.java  |  5 ++-
 .../java/org/apache/druid/cli/CliHistorical.java   | 18 +++-
 .../main/java/org/apache/druid/cli/CliIndexer.java | 12 -
 .../org/apache/druid/cli/CliMiddleManager.java | 28 +---
 23 files changed, 317 insertions(+), 103 deletions(-)
 copy 
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TaskAuditLogConfig.java
 => server/src/main/java/org/apache/druid/curator/ZkEnablementConfig.java (61%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (d0821de -> ee13630)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from d0821de  support for vectorizing expressions with non-existent inputs, 
more consistent type handling for non-vectorized expressions (#10499)
 add ee13630  optionally disable all of hardcoded zookeeper use (#9507)

No new revisions were added by this update.

Summary of changes:
 .../overlord/hrtr/HttpRemoteTaskRunner.java| 26 ---
 .../overlord/hrtr/HttpRemoteTaskRunnerFactory.java | 17 +--
 .../druid/indexing/worker/WorkerTaskManager.java   | 18 ++--
 .../worker/http/TaskManagementResource.java| 15 +++
 .../druid/indexing/worker/http/WorkerResource.java | 52 ++
 .../indexing/worker/http/WorkerResourceTest.java   |  6 ++-
 .../druid/cli/CliHistoricalForQueryRetryTest.java  |  1 +
 .../org/apache/druid/curator/CuratorModule.java|  7 ++-
 .../apache/druid/curator/ZkEnablementConfig.java   | 27 ++-
 .../druid/curator/discovery/DiscoveryModule.java   | 27 ---
 .../druid/curator/discovery/ServiceAnnouncer.java  | 16 +++
 .../org/apache/druid/guice/AnnouncerModule.java| 19 +++-
 .../coordination/BatchDataSegmentAnnouncer.java| 39 
 .../coordination/DataSegmentServerAnnouncer.java   | 16 +++
 .../druid/server/coordinator/DruidCoordinator.java | 28 
 .../server/coordinator/LoadQueueTaskMaster.java|  9 ++--
 .../druid/server/http/HistoricalResource.java  | 12 ++---
 .../coordinator/CuratorDruidCoordinatorTest.java   | 11 +++--
 .../server/coordinator/DruidCoordinatorTest.java   | 11 +++--
 .../java/org/apache/druid/cli/CliCoordinator.java  |  5 ++-
 .../java/org/apache/druid/cli/CliHistorical.java   | 18 +++-
 .../main/java/org/apache/druid/cli/CliIndexer.java | 12 -
 .../org/apache/druid/cli/CliMiddleManager.java | 28 +---
 23 files changed, 317 insertions(+), 103 deletions(-)
 copy 
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/TaskAuditLogConfig.java
 => server/src/main/java/org/apache/druid/curator/ZkEnablementConfig.java (61%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated: Disable sending server version in response headers (#9832)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 94226f1  Disable sending server version in response headers (#9832)
94226f1 is described below

commit 94226f1b3d0aadb3a77cabfd5c4c81c1c3388326
Author: Atul Mohan 
AuthorDate: Wed Sep 16 00:48:00 2020 -0500

Disable sending server version in response headers (#9832)

* Toggle sending of server version

* Remove config

Co-authored-by: Atul Mohan 
---
 .../org/apache/druid/server/initialization/jetty/JettyServerModule.java | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
 
b/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
index 39ccf8c..6fc105f 100644
--- 
a/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
+++ 
b/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
@@ -238,6 +238,7 @@ public class JettyServerModule extends JerseyServletModule
   }
 
   httpConfiguration.setRequestHeaderSize(config.getMaxRequestHeaderSize());
+  httpConfiguration.setSendServerVersion(false);
   final ServerConnector connector = new ServerConnector(server, new 
HttpConnectionFactory(httpConfiguration));
   if (node.isBindOnHost()) {
 connector.setHost(node.getHost());
@@ -323,6 +324,7 @@ public class JettyServerModule extends JerseyServletModule
   httpsConfiguration.setSecurePort(node.getTlsPort());
   httpsConfiguration.addCustomizer(new SecureRequestCustomizer());
   
httpsConfiguration.setRequestHeaderSize(config.getMaxRequestHeaderSize());
+  httpsConfiguration.setSendServerVersion(false);
   final ServerConnector connector = new ServerConnector(
   server,
   new SslConnectionFactory(sslContextFactory, HTTP_1_1_STRING),


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (0910d22 -> a607e9e)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 0910d22  Add SQL "OFFSET" clause. (#10279)
 add a607e9e  introduce interning of internal files names in 
SmooshedFileMapper (#10295)

No new revisions were added by this update.

Summary of changes:
 .../apache/druid/java/util/common/io/smoosh/SmooshedFileMapper.java | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (6cca724 -> 12ae841)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 6cca724  Add "offset" parameter to the Scan query. (#10233)
 add 12ae841  remove DruidLeaderClient.goAsync(..) that does not follow 
redirect. Replace its usage by DruidLeaderClient.go(..) with 
InputStreamFullResponseHandler (#9717)

No new revisions were added by this update.

Summary of changes:
 .../client/response/BytesFullResponseHolder.java   |   1 -
 .../http/client/response/FullResponseHolder.java   |   7 +-
 ...er.java => InputStreamFullResponseHandler.java} |  34 +++---
 ...der.java => InputStreamFullResponseHolder.java} |  33 +++---
 .../client/response/StringFullResponseHolder.java  |   1 -
 .../InputStreamFullResponseHandlerTest.java|  71 +
 .../org/apache/druid/client/DirectDruidClient.java |   3 +-
 .../apache/druid/client/JsonParserIterator.java|  18 +---
 .../apache/druid/discovery/DruidLeaderClient.java  |  27 +
 .../sql/calcite/schema/MetadataSegmentView.java|  47 ++---
 .../druid/sql/calcite/schema/SystemSchema.java | 117 ++---
 .../druid/sql/calcite/schema/SystemSchemaTest.java |  52 -
 .../druid/sql/calcite/util/CalciteTests.java   |   2 -
 13 files changed, 203 insertions(+), 210 deletions(-)
 copy 
core/src/main/java/org/apache/druid/java/util/http/client/response/{InputStreamResponseHandler.java
 => InputStreamFullResponseHandler.java} (54%)
 copy 
core/src/main/java/org/apache/druid/java/util/http/client/response/{StringFullResponseHolder.java
 => InputStreamFullResponseHolder.java} (64%)
 create mode 100644 
core/src/test/java/org/apache/druid/java/util/http/client/response/InputStreamFullResponseHandlerTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch 0.18.0 updated: druid-pac4j:add custom SSL handling to com.nimbusds.oauth2.sdk.http.HTTPRequest objects (#9695) (#9708)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch 0.18.0
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/0.18.0 by this push:
 new 0a8552a  druid-pac4j:add custom SSL handling to 
com.nimbusds.oauth2.sdk.http.HTTPRequest objects (#9695) (#9708)
0a8552a is described below

commit 0a8552ad12990df4157ebbc6f0755d75ec31c223
Author: Jihoon Son 
AuthorDate: Wed Apr 15 19:30:47 2020 -0700

druid-pac4j:add custom SSL handling to 
com.nimbusds.oauth2.sdk.http.HTTPRequest objects (#9695) (#9708)

Co-authored-by: Himanshu 
---
 extensions-core/druid-pac4j/pom.xml  | 12 +++-
 .../org/apache/druid/security/pac4j/Pac4jAuthenticator.java  | 10 ++
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/extensions-core/druid-pac4j/pom.xml 
b/extensions-core/druid-pac4j/pom.xml
index 9d7960b..577ee03 100644
--- a/extensions-core/druid-pac4j/pom.xml
+++ b/extensions-core/druid-pac4j/pom.xml
@@ -35,6 +35,10 @@
 
   
 3.8.3
+
+
+7.9
+6.5
   
 
   
@@ -60,10 +64,16 @@
   pac4j-oidc
   ${pac4j.version}
 
+
 
   com.nimbusds
   nimbus-jose-jwt
-  7.9
+  ${nimbus.jose.jwt.version}
+
+
+  com.nimbusds
+  oauth2-oidc-sdk
+  ${oauth2.oidc.sdk.version}
 
 
 
diff --git 
a/extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/Pac4jAuthenticator.java
 
b/extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/Pac4jAuthenticator.java
index c0473ce..2ca5000 100644
--- 
a/extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/Pac4jAuthenticator.java
+++ 
b/extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/Pac4jAuthenticator.java
@@ -25,7 +25,9 @@ import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeName;
 import com.google.common.base.Supplier;
 import com.google.common.base.Suppliers;
+import com.google.common.primitives.Ints;
 import com.google.inject.Provider;
+import com.nimbusds.oauth2.sdk.http.HTTPRequest;
 import org.apache.druid.server.security.AuthenticationResult;
 import org.apache.druid.server.security.Authenticator;
 import org.pac4j.core.config.Config;
@@ -130,7 +132,10 @@ public class Pac4jAuthenticator implements Authenticator
 oidcConf.setDiscoveryURI(oidcConfig.getDiscoveryURI());
 oidcConf.setExpireSessionWithToken(true);
 oidcConf.setUseNonce(true);
+
oidcConf.setReadTimeout(Ints.checkedCast(pac4jCommonConfig.getReadTimeout().getMillis()));
+
 oidcConf.setResourceRetriever(
+// ResourceRetriever is used to get Auth server configuration from 
"discoveryURI"
 new 
CustomSSLResourceRetriever(pac4jCommonConfig.getReadTimeout().getMillis(), 
sslSocketFactory)
 );
 
@@ -138,6 +143,11 @@ public class Pac4jAuthenticator implements Authenticator
 oidcClient.setUrlResolver(new DefaultUrlResolver(true));
 oidcClient.setCallbackUrlResolver(new NoParameterCallbackUrlResolver());
 
+// This is used by OidcClient in various places to make HTTPrequests.
+if (sslSocketFactory != null) {
+  HTTPRequest.setDefaultSSLSocketFactory(sslSocketFactory);
+}
+
 return new Config(Pac4jCallbackResource.SELF_URL, oidcClient);
   }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (332ca19 -> ca369e5)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 332ca19  Fix potential integer overflow issues (#9609)
 add ca369e5  druid-pac4j: add ability to use custom ssl trust store while 
talking to auth  server (#9637)

No new revisions were added by this update.

Summary of changes:
 docs/development/extensions-core/druid-pac4j.md|  5 +-
 extensions-core/druid-pac4j/pom.xml| 16 ++
 .../security/pac4j/CustomSSLResourceRetriever.java | 60 ++
 .../apache/druid/security/pac4j/OIDCConfig.java| 13 +
 .../druid/security/pac4j/Pac4jAuthenticator.java   | 24 +++--
 .../{OIDCConfig.java => Pac4jCommonConfig.java}| 44 ++--
 .../druid/security/pac4j/Pac4jDruidModule.java |  1 +
 .../druid/security/pac4j/OIDCConfigTest.java   | 28 +-
 .../security/pac4j/Pac4jCommonConfigTest.java  | 27 +-
 website/.spelling  |  2 +-
 10 files changed, 149 insertions(+), 71 deletions(-)
 create mode 100644 
extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/CustomSSLResourceRetriever.java
 copy 
extensions-core/druid-pac4j/src/main/java/org/apache/druid/security/pac4j/{OIDCConfig.java
 => Pac4jCommonConfig.java} (64%)
 copy 
processing/src/test/java/org/apache/druid/query/filter/FilterTuningTest.java => 
extensions-core/druid-pac4j/src/test/java/org/apache/druid/security/pac4j/OIDCConfigTest.java
 (57%)
 copy 
processing/src/test/java/org/apache/druid/query/filter/FilterTuningTest.java => 
extensions-core/druid-pac4j/src/test/java/org/apache/druid/security/pac4j/Pac4jCommonConfigTest.java
 (57%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (65de636 -> 8ccc0b2)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 65de636  Fix potential integer overflow in BufferArrayGrouper (#9605)
 add 8ccc0b2  Fix some flaws of KafkaEmitter (#9573)

No new revisions were added by this update.

Summary of changes:
 .../apache/druid/emitter/kafka/KafkaEmitter.java   | 30 --
 1 file changed, 10 insertions(+), 20 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (40e84a1 -> fc2897d)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 40e84a1  Eliminate common subfilters when converting it to a CNF 
(#9608)
 add fc2897d  pac4j: be noop if a previous authenticator in chain has 
successfully authenticated (#9620)

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/druid/security/pac4j/Pac4jFilter.java | 7 +++
 1 file changed, 7 insertions(+)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (40e84a1 -> fc2897d)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 40e84a1  Eliminate common subfilters when converting it to a CNF 
(#9608)
 add fc2897d  pac4j: be noop if a previous authenticator in chain has 
successfully authenticated (#9620)

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/druid/security/pac4j/Pac4jFilter.java | 7 +++
 1 file changed, 7 insertions(+)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated: Eliminate common subfilters when converting it to a CNF (#9608)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 40e84a1  Eliminate common subfilters when converting it to a CNF 
(#9608)
40e84a1 is described below

commit 40e84a171b8be8e87db20b0c5c189aef1b860f41
Author: Jihoon Son 
AuthorDate: Sun Apr 5 22:29:41 2020 -0700

Eliminate common subfilters when converting it to a CNF (#9608)
---
 .../druid/benchmark/FilterPartitionBenchmark.java  |   4 +-
 .../apache/druid/query/filter/BooleanFilter.java   |   5 +-
 .../apache/druid/query/filter/DimFilterUtils.java  |   4 +-
 .../apache/druid/query/filter/TrueDimFilter.java   |   8 +-
 .../segment/QueryableIndexStorageAdapter.java  |   9 +-
 .../org/apache/druid/segment/filter/AndFilter.java |  33 ++--
 .../org/apache/druid/segment/filter/Filters.java   |  56 +++---
 .../org/apache/druid/segment/filter/NotFilter.java |  32 +++-
 .../org/apache/druid/segment/filter/OrFilter.java  |  28 ++-
 .../apache/druid/segment/filter/TrueFilter.java|   9 +-
 .../segment/join/filter/JoinFilterAnalyzer.java|   8 +-
 .../druid/query/filter/AndDimFilterTest.java   |  29 ++-
 ...dDimFilterTest.java => DimFilterTestUtils.java} |  34 ++--
 .../apache/druid/query/filter/OrDimFilterTest.java |  53 +
 .../druid/segment/filter/BaseFilterTest.java   |   2 +-
 .../druid/segment/filter/FilterPartitionTest.java  |   4 +-
 .../filter/FilterTestUtils.java}   |  38 ++--
 .../apache/druid/segment/filter/FiltersTest.java   | 213 -
 ...tFilterTest.java => NotFilterEvaluateTest.java} |   6 +-
 .../apache/druid/segment/filter/NotFilterTest.java |  84 ++--
 20 files changed, 473 insertions(+), 186 deletions(-)

diff --git 
a/benchmarks/src/test/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
 
b/benchmarks/src/test/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
index b52cde1..547784e 100644
--- 
a/benchmarks/src/test/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
+++ 
b/benchmarks/src/test/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
@@ -377,7 +377,7 @@ public class FilterPartitionBenchmark
 Filter orFilter = new OrFilter(Arrays.asList(filter, filter2));
 
 StorageAdapter sa = new QueryableIndexStorageAdapter(qIndex);
-Sequence cursors = makeCursors(sa, Filters.convertToCNF(orFilter));
+Sequence cursors = makeCursors(sa, Filters.toCNF(orFilter));
 readCursors(cursors, blackhole);
   }
 
@@ -451,7 +451,7 @@ public class FilterPartitionBenchmark
 );
 
 StorageAdapter sa = new QueryableIndexStorageAdapter(qIndex);
-Sequence cursors = makeCursors(sa, 
Filters.convertToCNF(dimFilter3.toFilter()));
+Sequence cursors = makeCursors(sa, 
Filters.toCNF(dimFilter3.toFilter()));
 readCursors(cursors, blackhole);
   }
 
diff --git 
a/processing/src/main/java/org/apache/druid/query/filter/BooleanFilter.java 
b/processing/src/main/java/org/apache/druid/query/filter/BooleanFilter.java
index 9bbbdb6..e11153e 100644
--- a/processing/src/main/java/org/apache/druid/query/filter/BooleanFilter.java
+++ b/processing/src/main/java/org/apache/druid/query/filter/BooleanFilter.java
@@ -23,12 +23,13 @@ import org.apache.druid.segment.ColumnSelector;
 import org.apache.druid.segment.ColumnSelectorFactory;
 
 import java.util.HashSet;
-import java.util.List;
 import java.util.Set;
 
 public interface BooleanFilter extends Filter
 {
-  List getFilters();
+  ValueMatcher[] EMPTY_VALUE_MATCHER_ARRAY = new ValueMatcher[0];
+
+  Set getFilters();
 
   /**
* Get a ValueMatcher that applies this filter to row values.
diff --git 
a/processing/src/main/java/org/apache/druid/query/filter/DimFilterUtils.java 
b/processing/src/main/java/org/apache/druid/query/filter/DimFilterUtils.java
index 00a84fc..c980adb 100644
--- a/processing/src/main/java/org/apache/druid/query/filter/DimFilterUtils.java
+++ b/processing/src/main/java/org/apache/druid/query/filter/DimFilterUtils.java
@@ -52,9 +52,9 @@ public class DimFilterUtils
   static final byte COLUMN_COMPARISON_CACHE_ID = 0xD;
   static final byte EXPRESSION_CACHE_ID = 0xE;
   static final byte TRUE_CACHE_ID = 0xF;
-  public static byte BLOOM_DIM_FILTER_CACHE_ID = 0x10;
-  public static final byte STRING_SEPARATOR = (byte) 0xFF;
+  public static final byte BLOOM_DIM_FILTER_CACHE_ID = 0x10;
 
+  public static final byte STRING_SEPARATOR = (byte) 0xFF;
 
   static byte[] computeCacheKey(byte cacheIdKey, List filters)
   {
diff --git 
a/processing/src/main/java/org/apache/druid/query/filter/TrueDimFilter.java 
b/processing/src/main/java/org/apache/druid/query/filter/TrueDimFilter.java
index d10e6d9..2254358 100644
--- a/processing/src/main/java/org/apache/druid/query/filter/TrueDimFilter.java
+++ b/processing/src/main/java/org/apache/druid/query/filter/TrueDimFilter.java
@@ -20,9

[druid] branch 0.18.0 updated: Fix double count ssl connection metrics (#9594) (#9615)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch 0.18.0
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/0.18.0 by this push:
 new 404a558  Fix double count ssl connection metrics (#9594) (#9615)
404a558 is described below

commit 404a5586e44b18784d48204ff3372e561a36589c
Author: Clint Wylie 
AuthorDate: Sun Apr 5 15:01:43 2020 -0700

Fix double count ssl connection metrics (#9594) (#9615)

* fix double counted jetty/numOpenConnections metric for ssl connections

* tests

* more better

* style
---
 examples/bin/dsql-main |   2 +-
 .../docker/tls/generate-good-client-cert.sh|   2 +-
 .../tls/generate-server-certs-and-keystores.sh |   2 +-
 .../initialization/jetty/JettyServerModule.java|  15 +-
 .../druid/server/initialization/BaseJettyTest.java |  70 +++
 .../druid/server/initialization/JettyTest.java | 211 -
 server/src/test/resources/server.jks   | Bin 0 -> 1911 bytes
 server/src/test/resources/truststore.jks   | Bin 0 -> 1641 bytes
 8 files changed, 293 insertions(+), 9 deletions(-)

diff --git a/examples/bin/dsql-main b/examples/bin/dsql-main
index 8dfe882..cf68581 100755
--- a/examples/bin/dsql-main
+++ b/examples/bin/dsql-main
@@ -400,7 +400,7 @@ def main():
   parser_fmt.add_argument('--format', type=str, default='table', 
choices=('csv', 'tsv', 'json', 'table'), help='Result format')
   parser_fmt.add_argument('--header', action='store_true', help='Include 
header row for formats "csv" and "tsv"')
   parser_fmt.add_argument('--tsv-delimiter', type=str, default='\t', 
help='Delimiter for format "tsv"')
-  parser_oth.add_argument('--context-option', '-c', type=str, action='append', 
help='Set context option for this connection, see 
https://docs.imply.io/on-prem/query-data/sql for options')
+  parser_oth.add_argument('--context-option', '-c', type=str, action='append', 
help='Set context option for this connection, see 
https://druid.apache.org/docs/latest/querying/sql.html#connection-context for 
options')
   parser_oth.add_argument('--execute', '-e', type=str, help='Execute single 
SQL query')
   args = parser.parse_args()
 
diff --git a/integration-tests/docker/tls/generate-good-client-cert.sh 
b/integration-tests/docker/tls/generate-good-client-cert.sh
index 895e6c3..0f16c14 100755
--- a/integration-tests/docker/tls/generate-good-client-cert.sh
+++ b/integration-tests/docker/tls/generate-good-client-cert.sh
@@ -58,5 +58,5 @@ openssl x509 -req -days 3650 -in client.csr -CA root.pem 
-CAkey root.key -set_se
 openssl pkcs12 -export -in client.pem -inkey client.key -out client.p12 -name 
druid -CAfile root.pem -caname druid-it-root -password pass:druid123
 keytool -importkeystore -srckeystore client.p12 -srcstoretype PKCS12 
-destkeystore client.jks -deststoretype JKS -srcstorepass druid123 
-deststorepass druid123
 
-# Create a Java truststore with the imply test cluster root CA
+# Create a Java truststore with the druid test cluster root CA
 keytool -import -alias druid-it-root -keystore truststore.jks -file root.pem 
-storepass druid123 -noprompt
diff --git 
a/integration-tests/docker/tls/generate-server-certs-and-keystores.sh 
b/integration-tests/docker/tls/generate-server-certs-and-keystores.sh
index 8f38be3..e26cdac 100755
--- a/integration-tests/docker/tls/generate-server-certs-and-keystores.sh
+++ b/integration-tests/docker/tls/generate-server-certs-and-keystores.sh
@@ -63,7 +63,7 @@ openssl x509 -req -days 3650 -in server.csr -CA root.pem 
-CAkey root.key -set_se
 openssl pkcs12 -export -in server.pem -inkey server.key -out server.p12 -name 
druid -CAfile root.pem -caname druid-it-root -password pass:druid123
 keytool -importkeystore -srckeystore server.p12 -srcstoretype PKCS12 
-destkeystore server.jks -deststoretype JKS -srcstorepass druid123 
-deststorepass druid123
 
-# Create a Java truststore with the imply test cluster root CA
+# Create a Java truststore with the druid test cluster root CA
 keytool -import -alias druid-it-root -keystore truststore.jks -file root.pem 
-storepass druid123 -noprompt
 
 # Revoke one of the client certs
diff --git 
a/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
 
b/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
index bbb80e4..fc6f93e 100644
--- 
a/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
+++ 
b/server/src/main/java/org/apache/druid/server/initialization/jetty/JettyServerModule.java
@@ -22,6 +22,7 @@ package org.apache.druid.server.initialization.jetty;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider;
 import com.fasterxml.jackson.jaxrs

[druid] branch master updated: fix MAX_INTERMEDIATE_SIZE for DoubleMeanHolder (#9568)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 9081b5f  fix MAX_INTERMEDIATE_SIZE for DoubleMeanHolder (#9568)
9081b5f is described below

commit 9081b5f25c3d79e634ccc039a7d53e90cf43926d
Author: Stanislav Poryadnyi <37914083+ap0s...@users.noreply.github.com>
AuthorDate: Sat Mar 28 08:26:31 2020 +0300

fix MAX_INTERMEDIATE_SIZE for DoubleMeanHolder (#9568)

* fix MAX_INTERMEDIATE_SIZE for DoubleMeanHolder

* byte[] type handling in deserialize and finalizeComputation for 
DoubleMeanAggregatorFactory

* DoubleMeanAggregatorFactory tests: Max Intermediate Size, Deserialize, 
finalizeComputation

* moved byte[] check to first position

Co-authored-by: Stanislav 
---
 .../mean/DoubleMeanAggregatorFactory.java  |  8 ++-
 .../query/aggregation/mean/DoubleMeanHolder.java   | 12 ++--
 .../mean/DoubleMeanAggregatorFactoryTest.java  | 66 ++
 3 files changed, 78 insertions(+), 8 deletions(-)

diff --git 
a/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanAggregatorFactory.java
 
b/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanAggregatorFactory.java
index 5ed87be..d1da4d3 100644
--- 
a/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanAggregatorFactory.java
+++ 
b/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanAggregatorFactory.java
@@ -148,7 +148,9 @@ public class DoubleMeanAggregatorFactory extends 
AggregatorFactory
   @Override
   public Object deserialize(Object object)
   {
-if (object instanceof String) {
+if (object instanceof byte[]) {
+  return DoubleMeanHolder.fromBytes((byte[]) object);
+} else if (object instanceof String) {
   return 
DoubleMeanHolder.fromBytes(StringUtils.decodeBase64(StringUtils.toUtf8((String) 
object)));
 } else if (object instanceof DoubleMeanHolder) {
   return object;
@@ -161,7 +163,9 @@ public class DoubleMeanAggregatorFactory extends 
AggregatorFactory
   @Override
   public Object finalizeComputation(@Nullable Object object)
   {
-if (object instanceof DoubleMeanHolder) {
+if (object instanceof byte[]) {
+  return DoubleMeanHolder.fromBytes((byte[]) object).mean();
+} else if (object instanceof DoubleMeanHolder) {
   return ((DoubleMeanHolder) object).mean();
 } else if (object == null) {
   return null;
diff --git 
a/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanHolder.java
 
b/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanHolder.java
index f42c993..06aac1a 100644
--- 
a/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanHolder.java
+++ 
b/processing/src/main/java/org/apache/druid/query/aggregation/mean/DoubleMeanHolder.java
@@ -30,7 +30,7 @@ import java.util.Comparator;
 
 public class DoubleMeanHolder
 {
-  public static final int MAX_INTERMEDIATE_SIZE = Long.SIZE + Double.SIZE;
+  public static final int MAX_INTERMEDIATE_SIZE = Long.BYTES + Double.BYTES;
   public static final Comparator COMPARATOR = (o1, o2) -> 
Doubles.compare(o1.mean(), o2.mean());
 
   private double sum;
@@ -62,16 +62,16 @@ public class DoubleMeanHolder
 
   public byte[] toBytes()
   {
-ByteBuffer buf = ByteBuffer.allocate(Double.SIZE + Long.SIZE);
+ByteBuffer buf = ByteBuffer.allocate(Double.BYTES + Long.BYTES);
 buf.putDouble(0, sum);
-buf.putLong(Double.SIZE, count);
+buf.putLong(Double.BYTES, count);
 return buf.array();
   }
 
   public static DoubleMeanHolder fromBytes(byte[] data)
   {
 ByteBuffer buf = ByteBuffer.wrap(data);
-return new DoubleMeanHolder(buf.getDouble(0), buf.getLong(Double.SIZE));
+return new DoubleMeanHolder(buf.getDouble(0), buf.getLong(Double.BYTES));
   }
 
   public static void init(ByteBuffer buf, int position)
@@ -109,12 +109,12 @@ public class DoubleMeanHolder
 
   private static void writeCount(ByteBuffer buf, int position, long count)
   {
-buf.putLong(position + Double.SIZE, count);
+buf.putLong(position + Double.BYTES, count);
   }
 
   private static long getCount(ByteBuffer buf, int position)
   {
-return buf.getLong(position + Double.SIZE);
+return buf.getLong(position + Double.BYTES);
   }
 
   public static class Serializer extends JsonSerializer
diff --git 
a/processing/src/test/java/org/apache/druid/query/aggregation/mean/DoubleMeanAggregatorFactoryTest.java
 
b/processing/src/test/java/org/apache/druid/query/aggregation/mean/DoubleMeanAggregatorFactoryTest.java
new file mode 100644
index 000..0e49363
--- /dev/null
+++ 
b/processing/src/test/java/org/apache/druid/query/aggregation/mean/DoubleMeanAggregatorFactoryTest.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache

[druid] branch master updated (ff59d2e -> 1ba1a3c)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from ff59d2e  Move RowSignature from druid-sql to druid-processing and make 
use of it. (#9508)
 add 1ba1a3c  fix worker category on Indexer node (#9510)

No new revisions were added by this update.

Summary of changes:
 services/src/main/java/org/apache/druid/cli/CliIndexer.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (559c7b6 -> 75a5591)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from 559c7b6  Suppress CVEs for htrace-core4 and openstack-swift (#9489)
 add 75a5591  remove old unused zookeeper dependent lookups code (#9480)

No new revisions were added by this update.

Summary of changes:
 docs/querying/lookups.md   |   4 +-
 .../lookup/LookupListeningAnnouncerConfig.java |  11 +-
 .../apache/druid/query/lookup/LookupModule.java|  10 -
 .../lookup/LookupResourceListenerAnnouncer.java|  46 --
 .../announcer/ListenerResourceAnnouncer.java   | 111 -
 .../announcer/ListeningAnnouncerConfig.java| 101 -
 .../listener/resource/AbstractListenerHandler.java |   4 +-
 .../server/listener/resource/ListenerResource.java |   3 -
 .../announcer/ListenerResourceAnnouncerTest.java   | 134 --
 .../resource/AbstractListenerHandlerTest.java  | 277 
 .../listener/resource/ListenerResourceTest.java| 491 -
 11 files changed, 4 insertions(+), 1188 deletions(-)
 delete mode 100644 
server/src/main/java/org/apache/druid/query/lookup/LookupResourceListenerAnnouncer.java
 delete mode 100644 
server/src/main/java/org/apache/druid/server/listener/announcer/ListenerResourceAnnouncer.java
 delete mode 100644 
server/src/main/java/org/apache/druid/server/listener/announcer/ListeningAnnouncerConfig.java
 delete mode 100644 
server/src/test/java/org/apache/druid/server/listener/announcer/ListenerResourceAnnouncerTest.java
 delete mode 100644 
server/src/test/java/org/apache/druid/server/listener/resource/AbstractListenerHandlerTest.java
 delete mode 100644 
server/src/test/java/org/apache/druid/server/listener/resource/ListenerResourceTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[druid] branch master updated (f456d2f -> 072bbe2)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.


from f456d2f  Resource leak in DruidSegmentReader (#9476)
 add 072bbe2  remove ServerDiscoverySelector from DruidLeaderClient (#9481)

No new revisions were added by this update.

Summary of changes:
 .../apache/druid/discovery/DruidLeaderClient.java   | 19 +--
 .../druid/guice/CoordinatorDiscoveryModule.java | 19 ++-
 .../druid/guice/IndexingServiceDiscoveryModule.java | 19 ++-
 .../druid/discovery/DruidLeaderClientTest.java  | 21 ++---
 .../apache/druid/sql/calcite/util/CalciteTests.java |  5 +
 5 files changed, 12 insertions(+), 71 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (4510118 -> 9236dd9)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 4510118  HRTR: make pending task execution handling to go through all 
tasks on not finding worker slots (#8697)
 add 9236dd9  optionally enable Jetty ForwardedRequestCustomizer (#9010)

No new revisions were added by this update.

Summary of changes:
 docs/configuration/index.md|  1 +
 .../druid/server/initialization/ServerConfig.java  | 23 --
 .../jetty/CliIndexerServerModule.java  |  3 ++-
 .../initialization/jetty/JettyServerModule.java|  8 
 .../initialization/ServerConfigSerdeTest.java  |  5 -
 website/.spelling  |  2 ++
 6 files changed, 34 insertions(+), 8 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (810b85a -> 4510118)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 810b85a  allow druid.host to be undefined to use canonical hostname 
(#9019)
 add 4510118  HRTR: make pending task execution handling to go through all 
tasks on not finding worker slots (#8697)

No new revisions were added by this update.

Summary of changes:
 .../druid/java/util/emitter/EmittingLogger.java|   4 +-
 .../indexing/overlord/TaskRunnerWorkItem.java  |   1 +
 .../config/HttpRemoteTaskRunnerConfig.java |   8 -
 .../overlord/hrtr/HttpRemoteTaskRunner.java| 371 +++--
 .../hrtr/HttpRemoteTaskRunnerResource.java |  72 +++-
 .../druid/indexing/overlord/hrtr/WorkerHolder.java |  16 +
 .../overlord/hrtr/HttpRemoteTaskRunnerTest.java|   6 +
 7 files changed, 362 insertions(+), 116 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (24fe824 -> 8af41d7)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 24fe824  add readiness endpoints to processes having initialization 
delays (#8841)
 add 8af41d7  Update version to 0.18.0-incubating-SNAPSHOT (#9009)

No new revisions were added by this update.

Summary of changes:
 benchmarks/pom.xml   | 2 +-
 cloud/aws-common/pom.xml | 2 +-
 cloud/gcp-common/pom.xml | 2 +-
 core/pom.xml | 2 +-
 distribution/pom.xml | 2 +-
 extendedset/pom.xml  | 2 +-
 extensions-contrib/ambari-metrics-emitter/pom.xml| 2 +-
 extensions-contrib/azure-extensions/pom.xml  | 2 +-
 extensions-contrib/cassandra-storage/pom.xml | 2 +-
 extensions-contrib/cloudfiles-extensions/pom.xml | 2 +-
 extensions-contrib/distinctcount/pom.xml | 2 +-
 extensions-contrib/dropwizard-emitter/pom.xml| 2 +-
 extensions-contrib/graphite-emitter/pom.xml  | 2 +-
 extensions-contrib/influx-extensions/pom.xml | 2 +-
 extensions-contrib/influxdb-emitter/pom.xml  | 2 +-
 extensions-contrib/kafka-emitter/pom.xml | 2 +-
 extensions-contrib/materialized-view-maintenance/pom.xml | 2 +-
 extensions-contrib/materialized-view-selection/pom.xml   | 2 +-
 extensions-contrib/momentsketch/pom.xml  | 2 +-
 extensions-contrib/moving-average-query/pom.xml  | 2 +-
 extensions-contrib/opentsdb-emitter/pom.xml  | 2 +-
 extensions-contrib/redis-cache/pom.xml   | 2 +-
 extensions-contrib/sqlserver-metadata-storage/pom.xml| 2 +-
 extensions-contrib/statsd-emitter/pom.xml| 2 +-
 extensions-contrib/tdigestsketch/pom.xml | 2 +-
 extensions-contrib/thrift-extensions/pom.xml | 2 +-
 extensions-contrib/time-min-max/pom.xml  | 2 +-
 extensions-contrib/virtual-columns/pom.xml   | 2 +-
 extensions-core/avro-extensions/pom.xml  | 2 +-
 extensions-core/datasketches/pom.xml | 2 +-
 extensions-core/druid-basic-security/pom.xml | 2 +-
 extensions-core/druid-bloom-filter/pom.xml   | 2 +-
 extensions-core/druid-kerberos/pom.xml   | 2 +-
 extensions-core/ec2-extensions/pom.xml   | 2 +-
 extensions-core/google-extensions/pom.xml| 2 +-
 extensions-core/hdfs-storage/pom.xml | 2 +-
 extensions-core/histogram/pom.xml| 2 +-
 extensions-core/kafka-extraction-namespace/pom.xml   | 2 +-
 extensions-core/kafka-indexing-service/pom.xml   | 2 +-
 extensions-core/kinesis-indexing-service/pom.xml | 2 +-
 extensions-core/lookups-cached-global/pom.xml| 2 +-
 extensions-core/lookups-cached-single/pom.xml| 2 +-
 extensions-core/mysql-metadata-storage/pom.xml   | 2 +-
 extensions-core/orc-extensions/pom.xml   | 2 +-
 extensions-core/parquet-extensions/pom.xml   | 2 +-
 extensions-core/postgresql-metadata-storage/pom.xml  | 2 +-
 extensions-core/protobuf-extensions/pom.xml  | 2 +-
 extensions-core/s3-extensions/pom.xml| 2 +-
 extensions-core/simple-client-sslcontext/pom.xml | 2 +-
 extensions-core/stats/pom.xml| 2 +-
 hll/pom.xml  | 2 +-
 indexing-hadoop/pom.xml  | 2 +-
 indexing-service/pom.xml | 2 +-
 integration-tests/pom.xml| 2 +-
 pom.xml  | 2 +-
 processing/pom.xml   | 2 +-
 server/pom.xml   | 2 +-
 services/pom.xml | 2 +-
 sql/pom.xml  | 2 +-
 web-console/pom.xml  | 2 +-
 website/pom.xml  | 2 +-
 61 files changed, 61 insertions(+), 61 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (b4efaa6 -> 187cf0d)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from b4efaa6  unexclude necessary jackson mapper-asl jars (#8977)
 add 187cf0d  [Improvement] historical fast restart by lazy load columns 
metadata(20X faster) (#6988)

No new revisions were added by this update.

Summary of changes:
 docs/configuration/index.md|   1 +
 .../indexing/common/task/CompactionTaskTest.java   |  17 ++-
 .../AbstractMultiPhaseParallelIndexingTest.java|   2 +-
 .../java/org/apache/druid/segment/IndexIO.java | 131 ++---
 .../apache/druid/segment/SimpleQueryableIndex.java |  55 +
 .../MMappedQueryableSegmentizerFactory.java|   4 +-
 .../druid/segment/loading/SegmentizerFactory.java  |   2 +-
 .../druid/segment/loading/SegmentLoader.java   |   2 +-
 .../druid/segment/loading/SegmentLoaderConfig.java |   8 ++
 .../loading/SegmentLoaderLocalCacheManager.java|   4 +-
 .../org/apache/druid/server/SegmentManager.java|  10 +-
 .../coordination/SegmentLoadDropHandler.java   |   8 +-
 .../segment/loading/CacheTestSegmentLoader.java|   2 +-
 .../apache/druid/server/SegmentManagerTest.java|  28 ++---
 .../server/SegmentManagerThreadSafetyTest.java |   6 +-
 .../server/coordination/ServerManagerTest.java |   5 +-
 16 files changed, 179 insertions(+), 106 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (5c0fc0a -> 5adc821)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 5c0fc0a  Fix ambiguity about 
IndexerSQLMetadataStorageCoordinator.getUsedSegmentsForInterval() returning 
only non-overshadowed or all used segments (#8564)
 add 5adc821  add documentation for druid docker and k8s operator (#8802)

No new revisions were added by this update.

Summary of changes:
 .../cassandra.md => operations/kubernetes.md}  | 14 --
 website/.spelling  |  1 +
 2 files changed, 9 insertions(+), 6 deletions(-)
 copy docs/{development/extensions-contrib/cassandra.md => 
operations/kubernetes.md} (63%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (e70b71c -> 9c171e2)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from e70b71c  Fix verify script. (#8798)
 add 9c171e2  Message rejection absolute date (#8656)

No new revisions were added by this update.

Summary of changes:
 .../development/extensions-core/kafka-ingestion.md |   3 +-
 .../kafka/supervisor/KafkaSupervisorIOConfig.java  |   8 +-
 .../druid/indexing/kafka/KafkaSamplerSpecTest.java |   1 +
 .../supervisor/KafkaSupervisorIOConfigTest.java| 117 +
 .../kafka/supervisor/KafkaSupervisorTest.java  |   9 +-
 .../supervisor/KinesisSupervisorIOConfig.java  |   6 +-
 .../indexing/kinesis/KinesisSamplerSpecTest.java   |   1 +
 .../supervisor/KinesisSupervisorIOConfigTest.java  |   1 +
 .../kinesis/supervisor/KinesisSupervisorTest.java  |   4 +
 .../supervisor/SeekableStreamSupervisor.java   |  12 ++-
 .../SeekableStreamSupervisorIOConfig.java  |  23 +++-
 .../SeekableStreamSupervisorStateTest.java |   2 +-
 12 files changed, 152 insertions(+), 35 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (3abd0c4 -> a8b674e)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 3abd0c4  Web console: start transition to React hooks and functional 
components (#8722)
 add a8b674e  fix indexer when used with http inventory view (#8723)

No new revisions were added by this update.

Summary of changes:
 .../main/java/org/apache/druid/cli/CliIndexer.java | 24 +-
 1 file changed, 23 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (d54d2e1 -> 2c758ef)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from d54d2e1  Update segments.md (#8693)
 add 2c758ef  Support assign tasks to run on different categories of 
MiddleManagers (#7066)

No new revisions were added by this update.

Summary of changes:
 docs/configuration/index.md|  60 +++
 ...PendingTaskBasedWorkerProvisioningStrategy.java |   3 +-
 .../overlord/hrtr/HttpRemoteTaskRunner.java|   3 +-
 .../druid/indexing/overlord/hrtr/WorkerHolder.java |   3 +-
 ...utionWithCategorySpecWorkerSelectStrategy.java} |  39 ++---
 .../EqualDistributionWorkerSelectStrategy.java |   2 +-
 ...acityWithCategorySpecWorkerSelectStrategy.java} |  37 ++--
 .../setup/FillCapacityWorkerSelectStrategy.java|   2 +-
 .../overlord/setup/WorkerCategorySpec.java | 143 
 .../overlord/setup/WorkerSelectStrategy.java   |   4 +-
 .../indexing/overlord/setup/WorkerSelectUtils.java |  92 +-
 .../org/apache/druid/indexing/worker/Worker.java   |  18 +-
 .../druid/indexing/worker/http/WorkerResource.java |   3 +-
 .../indexing/overlord/ImmutableWorkerInfoTest.java |  50 --
 .../overlord/RemoteTaskRunnerTestUtils.java|  13 +-
 .../indexing/overlord/TaskRunnerUtilsTest.java |   3 +-
 .../overlord/WorkerTaskRunnerQueryAdpaterTest.java |   5 +-
 .../PendingTaskBasedProvisioningStrategyTest.java  |   3 +-
 .../SimpleProvisioningStrategyTest.java|   3 +-
 .../overlord/hrtr/HttpRemoteTaskRunnerTest.java|  35 ++--
 .../indexing/overlord/hrtr/WorkerHolderTest.java   |   3 +-
 ...butionWithAffinityWorkerSelectStrategyTest.java |  51 +++---
 ...onWithCategorySpecWorkerSelectStrategyTest.java | 190 +
 .../EqualDistributionWorkerSelectStrategyTest.java |  25 +--
 ...pacityWithAffinityWorkerSelectStrategyTest.java |  11 +-
 ...tyWithCategorySpecWorkerSelectStrategyTest.java | 190 +
 .../overlord/setup/WorkerCategorySpecTest.java |  49 +++---
 .../indexing/worker/WorkerTaskMonitorTest.java |   4 +-
 .../indexing/worker/http/WorkerResourceTest.java   |   4 +-
 .../apache/druid/discovery/WorkerNodeService.java  |  17 +-
 .../druid/indexing/worker/config/WorkerConfig.java |  11 ++
 .../druid/discovery/WorkerNodeServiceTest.java |   3 +-
 .../main/java/org/apache/druid/cli/CliIndexer.java |   6 +-
 .../org/apache/druid/cli/CliMiddleManager.java |   6 +-
 website/.spelling  |  10 ++
 35 files changed, 923 insertions(+), 178 deletions(-)
 copy 
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/{EqualDistributionWorkerSelectStrategy.java
 => EqualDistributionWithCategorySpecWorkerSelectStrategy.java} (65%)
 copy 
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/{FillCapacityWorkerSelectStrategy.java
 => FillCapacityWithCategorySpecWorkerSelectStrategy.java} (66%)
 create mode 100644 
indexing-service/src/main/java/org/apache/druid/indexing/overlord/setup/WorkerCategorySpec.java
 create mode 100644 
indexing-service/src/test/java/org/apache/druid/indexing/overlord/setup/EqualDistributionWithCategorySpecWorkerSelectStrategyTest.java
 create mode 100644 
indexing-service/src/test/java/org/apache/druid/indexing/overlord/setup/FillCapacityWithCategorySpecWorkerSelectStrategyTest.java
 copy core/src/test/java/org/apache/druid/js/JavaScriptConfigTest.java => 
indexing-service/src/test/java/org/apache/druid/indexing/overlord/setup/WorkerCategorySpecTest.java
 (51%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (0a20caf -> f0d74b2)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 0a20caf  update how to release doc (#8590)
 add f0d74b2  password provider for basic authentication of 
HttpEmitterConfig (#8618)

No new revisions were added by this update.

Summary of changes:
 .../druid/java/util/emitter/core/BaseHttpEmittingConfig.java  | 6 +++---
 .../apache/druid/java/util/emitter/core/HttpEmitterConfig.java| 3 ++-
 .../org/apache/druid/java/util/emitter/core/HttpPostEmitter.java  | 2 +-
 .../java/org/apache/druid/java/util/emitter/core/EmitterTest.java | 6 --
 .../druid/java/util/emitter/core/HttpEmitterConfigTest.java   | 8 
 .../java/util/emitter/core/ParametrizedUriEmitterConfigTest.java  | 4 ++--
 docs/configuration/index.md   | 2 +-
 7 files changed, 17 insertions(+), 14 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (82bfe86 -> 8537fbe)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 82bfe86  Make more package EverythingIsNonnullByDefault by default 
(#8198)
 add 8537fbe  Implementing dropwizard emitter for druid (#7363)

No new revisions were added by this update.

Summary of changes:
 .idea/misc.xml |   2 +-
 distribution/pom.xml   |   2 +
 docs/design/extensions-contrib/dropwizard.md   | 630 +
 .../pom.xml|  78 ++-
 .../emitter/dropwizard/DropwizardConverter.java}   |  56 +-
 .../emitter/dropwizard/DropwizardEmitter.java  | 200 +++
 .../dropwizard/DropwizardEmitterConfig.java| 139 +
 .../dropwizard/DropwizardEmitterModule.java}   |  45 +-
 .../emitter/dropwizard/DropwizardMetricSpec.java}  |  47 +-
 .../emitter/dropwizard/DropwizardReporter.java |  40 +-
 .../druid/emitter/dropwizard/GaugesCache.java  |  21 +-
 .../reporters/DropwizardConsoleReporter.java   | 143 +
 .../reporters/DropwizardJMXReporter.java   |  48 +-
 .../org.apache.druid.initialization.DruidModule|   2 +-
 .../main/resources/defaultMetricDimensions.json| 540 ++
 .../dropwizard/DropwizardEmitterConfigTest.java}   |  41 +-
 extensions-contrib/graphite-emitter/pom.xml|   2 +-
 pom.xml|   1 +
 website/.spelling  |   4 +
 19 files changed, 1860 insertions(+), 181 deletions(-)
 create mode 100644 docs/design/extensions-contrib/dropwizard.md
 copy extensions-contrib/{graphite-emitter => dropwizard-emitter}/pom.xml (80%)
 copy 
extensions-contrib/{statsd-emitter/src/main/java/org/apache/druid/emitter/statsd/DimensionConverter.java
 => 
dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardConverter.java}
 (57%)
 create mode 100644 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitter.java
 create mode 100644 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterConfig.java
 copy 
extensions-contrib/{graphite-emitter/src/main/java/org/apache/druid/emitter/graphite/GraphiteEmitterModule.java
 => 
dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterModule.java}
 (58%)
 copy 
extensions-contrib/{statsd-emitter/src/main/java/org/apache/druid/emitter/statsd/StatsDMetric.java
 => 
dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardMetricSpec.java}
 (58%)
 copy 
server/src/main/java/org/apache/druid/indexing/overlord/supervisor/SupervisorSpec.java
 => 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardReporter.java
 (61%)
 copy core/src/test/java/org/apache/druid/java/util/emitter/core/IntEvent.java 
=> 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/GaugesCache.java
 (73%)
 create mode 100644 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardConsoleReporter.java
 copy server/src/main/java/org/apache/druid/discovery/LookupNodeService.java => 
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardJMXReporter.java
 (57%)
 copy extensions-contrib/{graphite-emitter => 
dropwizard-emitter}/src/main/resources/META-INF/services/org.apache.druid.initialization.DruidModule
 (92%)
 create mode 100644 
extensions-contrib/dropwizard-emitter/src/main/resources/defaultMetricDimensions.json
 copy 
extensions-contrib/{opentsdb-emitter/src/test/java/org/apache/druid/emitter/opentsdb/OpentsdbEventTest.java
 => 
dropwizard-emitter/src/test/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterConfigTest.java}
 (53%)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch himanshug-patch-1 created (now 78e8efd)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch himanshug-patch-1
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


  at 78e8efd  CuratorDruidNodeDiscoveryProvider: do not ignore exception in 
listener execution and log it

This branch includes the following new commits:

 new 78e8efd  CuratorDruidNodeDiscoveryProvider: do not ignore exception in 
listener execution and log it

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.



-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] 01/01: CuratorDruidNodeDiscoveryProvider: do not ignore exception in listener execution and log it

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch himanshug-patch-1
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git

commit 78e8efdd1ec906ecbe1dc3103514feef6a6bc9dc
Author: Himanshu 
AuthorDate: Tue Oct 1 13:04:39 2019 -0700

CuratorDruidNodeDiscoveryProvider: do not ignore exception in listener 
execution and log it
---
 .../druid/curator/discovery/CuratorDruidNodeDiscoveryProvider.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git 
a/server/src/main/java/org/apache/druid/curator/discovery/CuratorDruidNodeDiscoveryProvider.java
 
b/server/src/main/java/org/apache/druid/curator/discovery/CuratorDruidNodeDiscoveryProvider.java
index 938dd1b..93903de 100644
--- 
a/server/src/main/java/org/apache/druid/curator/discovery/CuratorDruidNodeDiscoveryProvider.java
+++ 
b/server/src/main/java/org/apache/druid/curator/discovery/CuratorDruidNodeDiscoveryProvider.java
@@ -330,7 +330,7 @@ public class CuratorDruidNodeDiscoveryProvider extends 
DruidNodeDiscoveryProvide
   runnable.run();
 }
 catch (Exception ex) {
-  log.error(errMsgFormat, args);
+  log.error(ex, errMsgFormat, args);
 }
   });
 }


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: bug fix for lookup leak when we remove the last lookup from lookup tier (#8598)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new e1b4a3a  bug fix for lookup leak when we remove the last lookup from 
lookup tier (#8598)
e1b4a3a is described below

commit e1b4a3ab7162e74ab86465f90ad50c72c1c19f15
Author: Faxian Zhao <232775...@qq.com>
AuthorDate: Fri Sep 27 18:55:02 2019 +0800

bug fix for lookup leak when we remove the last lookup from lookup tier 
(#8598)

* bug fix for lookup leak when we remove the last lookup from lookup tier

* warnings about lookups that will never be loaded

* fix unit test
---
 .../lookup/cache/LookupCoordinatorManager.java  | 21 -
 .../lookup/cache/LookupCoordinatorManagerTest.java  |  4 
 2 files changed, 20 insertions(+), 5 deletions(-)

diff --git 
a/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
 
b/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
index b6589aa..cd3ac2c 100644
--- 
a/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
+++ 
b/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
@@ -557,16 +557,27 @@ public class LookupCoordinatorManager
 
 try {
   List> futures = new ArrayList<>();
-  for (Map.Entry> 
tierEntry : allLookupTiers.entrySet()) {
 
-LOG.debug("Starting lookup mgmt for tier [%s].", tierEntry.getKey());
+  Set discoveredLookupTiers = lookupNodeDiscovery.getAllTiers();
 
-final Map tierLookups = 
tierEntry.getValue();
-for (final HostAndPortWithScheme node : 
lookupNodeDiscovery.getNodesInTier(tierEntry.getKey())) {
+  // Check and Log warnings about lookups configured by user in DB but no 
nodes discovered to load those.
+  for (String tierInDB : allLookupTiers.keySet()) {
+if (!discoveredLookupTiers.contains(tierInDB) &&
+!allLookupTiers.getOrDefault(tierInDB, 
ImmutableMap.of()).isEmpty()) {
+  LOG.warn("Found lookups for tier [%s] in DB, but no nodes discovered 
for it", tierInDB);
+}
+  }
+
+  for (String tier : discoveredLookupTiers) {
+
+LOG.debug("Starting lookup mgmt for tier [%s].", tier);
+
+final Map tierLookups = 
allLookupTiers.getOrDefault(tier, ImmutableMap.of());
+for (final HostAndPortWithScheme node : 
lookupNodeDiscovery.getNodesInTier(tier)) {
 
   LOG.debug(
   "Starting lookup mgmt for tier [%s] and host [%s:%s:%s].",
-  tierEntry.getKey(),
+  tier,
   node.getScheme(),
   node.getHostText(),
   node.getPort()
diff --git 
a/server/src/test/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManagerTest.java
 
b/server/src/test/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManagerTest.java
index 772b185..962c950 100644
--- 
a/server/src/test/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManagerTest.java
+++ 
b/server/src/test/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManagerTest.java
@@ -1139,6 +1139,10 @@ public class LookupCoordinatorManagerTest
 
 EasyMock.reset(lookupNodeDiscovery);
 EasyMock
+.expect(lookupNodeDiscovery.getAllTiers())
+.andReturn(ImmutableSet.of("tier1"))
+.once();
+EasyMock
 .expect(lookupNodeDiscovery.getNodesInTier("tier1"))
 .andReturn(ImmutableList.of(host1, host2))
 .anyTimes();


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (9f1f5e1 -> a280c5d)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 9f1f5e1  doubleMean aggregator to be used at query time (#8459)
 add a280c5d  fix queuedSize not decrease in HttpLoadQueuePeon when load 
failed (#8596)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/druid/server/coordinator/HttpLoadQueuePeon.java | 7 +++
 1 file changed, 7 insertions(+)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: doubleMean aggregator to be used at query time (#8459)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 9f1f5e1  doubleMean aggregator to be used at query time (#8459)
9f1f5e1 is described below

commit 9f1f5e115c0595e208c0f59e12bf2a5e26ed0c29
Author: Himanshu 
AuthorDate: Thu Sep 26 08:04:33 2019 -0700

doubleMean aggregator to be used at query time (#8459)

* doubleMean aggregator for computing mean

* make docs

* build fixes

* address review comment: handle null args
---
 .../druid/java/util/common/collect/Utils.java  |   9 +
 docs/querying/aggregations.md  |   8 +
 .../apache/druid/jackson/AggregatorsModule.java|   5 +
 .../druid/query/aggregation/AggregatorUtil.java|   2 +
 .../aggregation/SimpleDoubleBufferAggregator.java  |   2 +-
 .../aggregation/mean/DoubleMeanAggregator.java |  86 ++
 .../mean/DoubleMeanAggregatorFactory.java  | 181 +
 .../mean/DoubleMeanBufferAggregator.java   | 101 
 .../query/aggregation/mean/DoubleMeanHolder.java   | 136 
 .../mean/DoubleMeanVectorAggregator.java   |  81 +
 .../druid/query/groupby/GroupByQueryConfig.java|   2 +-
 .../query/aggregation/AggregationTestHelper.java   |   2 +-
 .../mean/DoubleMeanAggregationTest.java| 159 ++
 .../query/aggregation/mean/SimpleTestIndex.java| 113 +
 14 files changed, 884 insertions(+), 3 deletions(-)

diff --git 
a/core/src/main/java/org/apache/druid/java/util/common/collect/Utils.java 
b/core/src/main/java/org/apache/druid/java/util/common/collect/Utils.java
index de7a239..429660f 100644
--- a/core/src/main/java/org/apache/druid/java/util/common/collect/Utils.java
+++ b/core/src/main/java/org/apache/druid/java/util/common/collect/Utils.java
@@ -81,4 +81,13 @@ public class Utils
 
 return true;
   }
+
+  public static String safeObjectClassGetName(@Nullable Object o)
+  {
+if (o == null) {
+  return "NULL";
+} else {
+  return o.getClass().getName();
+}
+  }
 }
diff --git a/docs/querying/aggregations.md b/docs/querying/aggregations.md
index 3f63095..4bc7605 100644
--- a/docs/querying/aggregations.md
+++ b/docs/querying/aggregations.md
@@ -120,6 +120,14 @@ Computes and stores the sum of values as 32-bit floating 
point value. Similar to
 { "type" : "longMax", "name" : , "fieldName" :  }
 ```
 
+### `doubleMean` aggregator
+
+Computes and returns arithmetic mean of a column values as 64 bit float value. 
This is a query time aggregator only and should not be used during indexing.
+
+```json
+{ "type" : "doubleMean", "name" : , "fieldName" :  }
+```
+
 ### First / Last aggregator
 
 (Double/Float/Long) First and Last aggregator cannot be used in ingestion 
spec, and should only be specified as part of queries.
diff --git 
a/processing/src/main/java/org/apache/druid/jackson/AggregatorsModule.java 
b/processing/src/main/java/org/apache/druid/jackson/AggregatorsModule.java
index 6689d27..aba52a0 100644
--- a/processing/src/main/java/org/apache/druid/jackson/AggregatorsModule.java
+++ b/processing/src/main/java/org/apache/druid/jackson/AggregatorsModule.java
@@ -53,6 +53,8 @@ import 
org.apache.druid.query.aggregation.last.FloatLastAggregatorFactory;
 import org.apache.druid.query.aggregation.last.LongLastAggregatorFactory;
 import org.apache.druid.query.aggregation.last.StringLastAggregatorFactory;
 import 
org.apache.druid.query.aggregation.last.StringLastFoldingAggregatorFactory;
+import org.apache.druid.query.aggregation.mean.DoubleMeanAggregatorFactory;
+import org.apache.druid.query.aggregation.mean.DoubleMeanHolder;
 import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
 import org.apache.druid.query.aggregation.post.ConstantPostAggregator;
 import org.apache.druid.query.aggregation.post.DoubleGreatestPostAggregator;
@@ -79,6 +81,8 @@ public class AggregatorsModule extends SimpleModule
 
 setMixInAnnotation(AggregatorFactory.class, AggregatorFactoryMixin.class);
 setMixInAnnotation(PostAggregator.class, PostAggregatorMixin.class);
+
+addSerializer(DoubleMeanHolder.class, 
DoubleMeanHolder.Serializer.INSTANCE);
   }
 
   @JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@@ -105,6 +109,7 @@ public class AggregatorsModule extends SimpleModule
   @JsonSubTypes.Type(name = "stringFirstFold", value = 
StringFirstFoldingAggregatorFactory.class),
   @JsonSubTypes.Type(name = "longLast", value = 
LongLastAggregatorFactory.class),
   @JsonSubTypes.Type(name = "doubleLast", value = 
DoubleLastAggregatorFactory.class),
+  @JsonSubTypes.Type(name = "doubleMean

[incubator-druid] branch master updated (563718c -> a75ddaa)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 563718c  add druid version correctly (#8586)
 add a75ddaa  Add TrustedDomain Authenticator (#8248)

No new revisions were added by this update.

Summary of changes:
 docs/design/auth.md|  28 
 .../initialization/jetty/ServletFilterHolder.java  |   2 +
 .../apache/druid/server/security/AuthConfig.java   |   2 +
 .../druid/server/security/Authenticator.java   |   1 +
 ...icator.java => TrustedDomainAuthenticator.java} |  66 +++--
 .../security/TrustedDomainAuthenticatorTest.java   | 164 +
 website/.spelling  |   1 +
 7 files changed, 252 insertions(+), 12 deletions(-)
 copy 
server/src/main/java/org/apache/druid/server/security/{AnonymousAuthenticator.java
 => TrustedDomainAuthenticator.java} (52%)
 create mode 100644 
server/src/test/java/org/apache/druid/server/security/TrustedDomainAuthenticatorTest.java


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (e184d24 -> 62afbca)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from e184d24  add support for dogstatsd events in statsd-emitter (#8546)
 add 62afbca  update HRTR to account for task known to be running on a 
worker when it shows up (#8427)

No new revisions were added by this update.

Summary of changes:
 .../overlord/hrtr/HttpRemoteTaskRunner.java| 38 ++--
 .../druid/indexing/overlord/hrtr/WorkerHolder.java | 11 +++-
 .../overlord/hrtr/HttpRemoteTaskRunnerTest.java| 67 ++
 .../indexing/overlord/hrtr/WorkerHolderTest.java   | 28 +++--
 4 files changed, 107 insertions(+), 37 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (2b04c22 -> 58e2634)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 2b04c22  split segment view query into two (#8485)
 add 58e2634  Update RoaringBitmap version to 0.8.11 (#8490)

No new revisions were added by this update.

Summary of changes:
 licenses.yaml | 2 +-
 pom.xml   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (bba391a -> ec836ae)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from bba391a  Fix alerts from LGTM about python files (#8383)
 add ec836ae  Fix result of division may be truncated (#8355)

No new revisions were added by this update.

Summary of changes:
 examples/bin/dsql-main | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (ec836ae -> 9fa3407)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from ec836ae  Fix result of division may be truncated (#8355)
 add 9fa3407  Suppress index-out-of-bounds warning from LGTM about loop 
unrolling (#8380)

No new revisions were added by this update.

Summary of changes:
 .../druid/query/topn/TopNLexicographicResultBuilder.java | 16 +---
 .../druid/query/topn/TopNNumericResultBuilder.java   | 16 +---
 2 files changed, 18 insertions(+), 14 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (1fe4ecf -> bba391a)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 1fe4ecf  StringDictionaryEncodedColumn dimSelector to return 
CARDINALITY_UNKNOWN with extractionFn (#8433)
 add bba391a  Fix alerts from LGTM about python files (#8383)

No new revisions were added by this update.

Summary of changes:
 distribution/bin/check-licenses.py  | 1 -
 distribution/bin/find-missing-backports.py  | 6 ++
 distribution/bin/generate-license-dependency-reports.py | 9 +++--
 distribution/bin/jar-notice-lister.py   | 6 +++---
 4 files changed, 8 insertions(+), 14 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (645799f -> 1fe4ecf)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 645799f  disallow whitespace characters except space in data source 
names (#8465)
 add 1fe4ecf  StringDictionaryEncodedColumn dimSelector to return 
CARDINALITY_UNKNOWN with extractionFn (#8433)

No new revisions were added by this update.

Summary of changes:
 .../org/apache/druid/segment/DimensionDictionarySelector.java | 7 ++-
 .../druid/segment/column/StringDictionaryEncodedColumn.java   | 8 
 2 files changed, 14 insertions(+), 1 deletion(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: disallow whitespace characters except space in data source names (#8465)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 645799f  disallow whitespace characters except space in data source 
names (#8465)
645799f is described below

commit 645799f97702e5fe2ed61d97839388c298b5ec79
Author: Rye 
AuthorDate: Fri Sep 6 08:55:21 2019 -0700

disallow whitespace characters except space in data source names (#8465)

* disallow whitespace characters in data source names

* wrapped preconditions in a function, and simplify unit tests code

* Fixed regex to allow space, simplified repeat logic

* Fixed import style against mvn checkstyle

* Add msg in case test fails, use emptyMap(), improved naming

* Changes on assertion functions

* change wording of "whitespace" to "whitespace except space" to avoid 
misleading
---
 .../apache/druid/segment/indexing/DataSchema.java  | 23 +++--
 .../druid/segment/indexing/DataSchemaTest.java | 40 +-
 2 files changed, 59 insertions(+), 4 deletions(-)

diff --git 
a/server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java 
b/server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java
index 10192c7..6ce4f0d 100644
--- a/server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java
+++ b/server/src/main/java/org/apache/druid/segment/indexing/DataSchema.java
@@ -41,13 +41,17 @@ import java.util.Arrays;
 import java.util.HashSet;
 import java.util.Map;
 import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
 
 /**
+ *
  */
 public class DataSchema
 {
   private static final Logger log = new Logger(DataSchema.class);
-
+  private static final Pattern INVALIDCHARS = Pattern.compile("(?s).*[^\\S 
].*");
   private final String dataSource;
   private final Map parser;
   private final AggregatorFactory[] aggregators;
@@ -72,8 +76,7 @@ public class DataSchema
 this.parser = parser;
 this.transformSpec = transformSpec == null ? TransformSpec.NONE : 
transformSpec;
 
-Preconditions.checkArgument(!Strings.isNullOrEmpty(dataSource), 
"dataSource cannot be null or empty. Please provide a dataSource.");
-Preconditions.checkArgument(!dataSource.contains("/"), "dataSource cannot 
contain the '/' character.");
+validateDatasourceName(dataSource);
 this.dataSource = dataSource;
 
 if (granularitySpec == null) {
@@ -98,6 +101,20 @@ public class DataSchema
 this.aggregators = aggregators == null ? new AggregatorFactory[]{} : 
aggregators;
   }
 
+  static void validateDatasourceName(String dataSource)
+  {
+Preconditions.checkArgument(
+!Strings.isNullOrEmpty(dataSource),
+"dataSource cannot be null or empty. Please provide a dataSource."
+);
+Matcher m = INVALIDCHARS.matcher(dataSource);
+Preconditions.checkArgument(
+!m.matches(),
+"dataSource cannot contain whitespace character except space."
+);
+Preconditions.checkArgument(!dataSource.contains("/"), "dataSource cannot 
contain the '/' character.");
+  }
+
   @JsonProperty
   public String getDataSource()
   {
diff --git 
a/server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java 
b/server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java
index c059f29..437efd3 100644
--- a/server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java
+++ b/server/src/test/java/org/apache/druid/segment/indexing/DataSchemaTest.java
@@ -22,6 +22,7 @@ package org.apache.druid.segment.indexing;
 import com.fasterxml.jackson.databind.JsonMappingException;
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
 import org.apache.druid.data.input.InputRow;
 import org.apache.druid.data.input.impl.DimensionsSpec;
@@ -52,6 +53,7 @@ import java.io.IOException;
 import java.nio.ByteBuffer;
 import java.nio.charset.StandardCharsets;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.Map;
 
 public class DataSchemaTest
@@ -286,7 +288,7 @@ public class DataSchemaTest
 // Jackson creates a default type parser (StringInputRowParser) for an 
invalid type.
 schema.getParser();
   }
-  
+
   @Test
   public void testEmptyDatasource()
   {
@@ -324,6 +326,42 @@ public class DataSchemaTest
 );
   }
 
+
+  @Test
+  public void testInvalidWhitespaceDatasource()
+  {
+Map invalidCharToDataSourceName = ImmutableMap.of(
+"\\t", "\tab\t",
+"\\r", "\rcarriage\return\r",
+"\\n", "\nnew\nline\n&qu

[incubator-druid] branch master updated (637a9e8 -> ee4ebb4)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 637a9e8  Web console: Fix segment re-ingest (#8454)
 add ee4ebb4  make single/multi value string column handling official in 
aggregation (#8428)

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/druid/query/aggregation/AggregatorFactory.java  | 6 ++
 1 file changed, 6 insertions(+)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (4d87a19 -> d5d170f)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 4d87a19  Logging emitter to publish query and other metric events as 
valid json objects (#8359)
 add d5d170f  skip unnecessary aggregate(..) calls with 
LimitedBufferHashGrouper (#8412)

No new revisions were added by this update.

Summary of changes:
 .../epinephelinae/AbstractBufferHashGrouper.java   |  5 ++--
 .../groupby/epinephelinae/BufferHashGrouper.java   |  2 +-
 .../epinephelinae/LimitedBufferHashGrouper.java| 34 ++
 3 files changed, 19 insertions(+), 22 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (44dd5b5 -> 4d87a19)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 44dd5b5  add jaxb-runtime to fix exception with newer versions of java 
(#8409)
 add 4d87a19  Logging emitter to publish query and other metric events as 
valid json objects (#8359)

No new revisions were added by this update.

Summary of changes:
 .../druid/java/util/emitter/core/LoggingEmitter.java | 11 +--
 .../util/emitter/service/ServiceMetricEvent.java |  2 +-
 .../DefaultMovingAverageQueryMetrics.java|  7 ---
 .../DefaultMovingAverageQueryMetricsFactory.java | 16 ++--
 .../druid/indexing/kafka/KafkaIndexTaskTest.java |  3 +--
 .../query/DefaultGenericQueryMetricsFactory.java | 16 ++--
 .../org/apache/druid/query/DefaultQueryMetrics.java  | 20 ++--
 .../query/groupby/DefaultGroupByQueryMetrics.java|  7 ---
 .../groupby/DefaultGroupByQueryMetricsFactory.java   | 16 ++--
 .../timeseries/DefaultTimeseriesQueryMetrics.java|  6 --
 .../DefaultTimeseriesQueryMetricsFactory.java| 16 ++--
 .../druid/query/topn/DefaultTopNQueryMetrics.java|  7 ---
 .../query/topn/DefaultTopNQueryMetricsFactory.java   | 16 ++--
 .../apache/druid/query/DefaultQueryMetricsTest.java  |  5 ++---
 .../groupby/DefaultGroupByQueryMetricsTest.java  |  5 ++---
 .../DefaultTimeseriesQueryMetricsTest.java   |  5 ++---
 .../query/topn/DefaultTopNQueryMetricsTest.java  |  5 ++---
 .../server/log/EmittingRequestLoggerProvider.java|  2 +-
 .../realtime/appenderator/AppenderatorTester.java|  3 +--
 .../server/AsyncQueryForwardingServletTest.java  |  4 ++--
 .../org/apache/druid/server/QueryResourceTest.java   | 16 
 .../druid/server/coordination/ServerManagerTest.java |  2 +-
 .../apache/druid/sql/calcite/util/CalciteTests.java  |  4 ++--
 23 files changed, 42 insertions(+), 152 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (b2821a8 -> a4b842a)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from b2821a8  do not exclude client core jar (#8339)
 add a4b842a  Speed up ExpressionSelectors.makeExprEvalSelector (#8373)

No new revisions were added by this update.

Summary of changes:
 .../main/java/org/apache/druid/math/expr/Expr.java | 68 +-
 .../org/apache/druid/math/expr/ExprMacroTable.java | 21 ++-
 .../org/apache/druid/math/expr/ParserTest.java |  4 +-
 3 files changed, 63 insertions(+), 30 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (4416169 -> 167a0e7)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 4416169  Put readme in a good order (#8372)
 add 167a0e7  [opentsdb-emitter] Fix ingest persists metrics typo (#8233)

No new revisions were added by this update.

Summary of changes:
 .../opentsdb-emitter/src/main/resources/defaultMetrics.json| 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: make double sum/min/max agg work on string columns (#8243)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 176da53  make double sum/min/max agg work on string columns (#8243)
176da53 is described below

commit 176da53996220776979f4ef7672ca4d31acb00b6
Author: Himanshu 
AuthorDate: Tue Aug 13 15:55:14 2019 -0700

make double sum/min/max agg work on string columns (#8243)

* make double sum/min/max agg work on string columns

* style and compilation fixes

* fix tests

* address review comments

* add comment on SimpleDoubleAggregatorFactory

* make checkstyle happy
---
 .../org/apache/druid/java/util/common/Numbers.java |  24 +++
 .../druid/query/aggregation/AggregatorUtil.java|   3 +-
 .../query/aggregation/DelegatingAggregator.java|  74 
 .../aggregation/DelegatingBufferAggregator.java|  95 ++
 .../aggregation/DoubleMaxAggregatorFactory.java|  15 +-
 .../aggregation/DoubleMinAggregatorFactory.java|  15 +-
 .../aggregation/DoubleSumAggregatorFactory.java|  28 ++-
 .../aggregation/SimpleDoubleAggregatorFactory.java |  62 ++-
 .../StringColumnDoubleAggregatorWrapper.java   |  69 +++
 .../StringColumnDoubleBufferAggregatorWrapper.java |  70 +++
 .../SettableValueDoubleColumnValueSelector.java|  56 ++
 .../apache/druid/query/SchemaEvolutionTest.java|   8 +-
 .../query/aggregation/AggregationTestHelper.java   |  54 ++
 .../aggregation/DoubleMaxAggregationTest.java  |   1 +
 .../aggregation/DoubleMinAggregationTest.java  |   1 +
 .../aggregation/StringColumnAggregationTest.java   | 201 +
 16 files changed, 728 insertions(+), 48 deletions(-)

diff --git a/core/src/main/java/org/apache/druid/java/util/common/Numbers.java 
b/core/src/main/java/org/apache/druid/java/util/common/Numbers.java
index c9d40d7..c40cb8e 100644
--- a/core/src/main/java/org/apache/druid/java/util/common/Numbers.java
+++ b/core/src/main/java/org/apache/druid/java/util/common/Numbers.java
@@ -19,6 +19,10 @@
 
 package org.apache.druid.java.util.common;
 
+import com.google.common.primitives.Doubles;
+
+import javax.annotation.Nullable;
+
 public final class Numbers
 {
   /**
@@ -92,6 +96,26 @@ public final class Numbers
 }
   }
 
+  /**
+   * Try parsing the given Number or String object val as double.
+   * @param val
+   * @param nullValue value to return when input was string type but not 
parseable into double value
+   * @return parsed double value
+   */
+  public static double tryParseDouble(@Nullable Object val, double nullValue)
+  {
+if (val == null) {
+  return nullValue;
+} else if (val instanceof Number) {
+  return ((Number) val).doubleValue();
+} else if (val instanceof String) {
+  Double d = Doubles.tryParse((String) val);
+  return d == null ? nullValue : d.doubleValue();
+} else {
+  throw new IAE("Unknown object type [%s]", val.getClass().getName());
+}
+  }
+
   public static int toIntExact(long value, String error)
   {
 if ((int) value != value) {
diff --git 
a/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
 
b/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
index ca0eb3a..13a16d0 100644
--- 
a/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
+++ 
b/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
@@ -25,7 +25,6 @@ import org.apache.druid.java.util.common.Pair;
 import org.apache.druid.math.expr.Expr;
 import org.apache.druid.math.expr.ExprEval;
 import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
-import org.apache.druid.segment.BaseDoubleColumnValueSelector;
 import org.apache.druid.segment.BaseFloatColumnValueSelector;
 import org.apache.druid.segment.BaseLongColumnValueSelector;
 import org.apache.druid.segment.ColumnSelectorFactory;
@@ -266,7 +265,7 @@ public class AggregatorUtil
   /**
* Only one of fieldName and fieldExpression should be non-null
*/
-  static BaseDoubleColumnValueSelector 
makeColumnValueSelectorWithDoubleDefault(
+  static ColumnValueSelector makeColumnValueSelectorWithDoubleDefault(
   final ColumnSelectorFactory metricFactory,
   @Nullable final String fieldName,
   @Nullable final Expr fieldExpression,
diff --git 
a/processing/src/main/java/org/apache/druid/query/aggregation/DelegatingAggregator.java
 
b/processing/src/main/java/org/apache/druid/query/aggregation/DelegatingAggregator.java
new file mode 100644
index 000..c1b4b40
--- /dev/null
+++ 
b/processing/src/main/java/org/apache/druid/query/aggregation/DelegatingAggregator.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See 

[incubator-druid] branch master updated: Update docker build (#8244)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 38b6047  Update docker build (#8244)
38b6047 is described below

commit 38b6047aa96e2ea3a6651aed138ee46566d596f6
Author: ilhanadiyaman 
AuthorDate: Tue Aug 6 17:32:59 2019 +0200

Update docker build (#8244)
---
 distribution/docker/Dockerfile |  6 -
 distribution/docker/README.md  |  6 ++---
 distribution/docker/druid.sh   | 55 +-
 3 files changed, 41 insertions(+), 26 deletions(-)

diff --git a/distribution/docker/Dockerfile b/distribution/docker/Dockerfile
index fa80ba2..9db493c 100644
--- a/distribution/docker/Dockerfile
+++ b/distribution/docker/Dockerfile
@@ -19,9 +19,13 @@
 
 FROM maven:3-jdk-8 as builder
 
+RUN apt-get update && apt-get install --no-install-recommends -y python3-pip 
python3-setuptools python3-wheel\
+  && rm -rf /var/lib/apt/lists/*
+RUN pip3 install --no-cache-dir pyyaml
+
 COPY . /src
 WORKDIR /src
-RUN mvn install -ff -DskipTests -Dforbiddenapis.skip=true -Pdist 
-Pbundle-contrib-exts
+RUN mvn dependency:go-offline install -ff -q -B -DskipTests 
-Dforbiddenapis.skip=true -Pdist -Pbundle-contrib-exts
 
 RUN \
  VER=$(mvn -B org.apache.maven.plugins:maven-help-plugin:3.1.1:evaluate 
-Dexpression=project.version -q -DforceStdout=true -f pom.xml 2>/dev/null) \
diff --git a/distribution/docker/README.md b/distribution/docker/README.md
index b029522..64f51d6 100644
--- a/distribution/docker/README.md
+++ b/distribution/docker/README.md
@@ -23,9 +23,9 @@ From the root of the repo, run `docker build -t druid:tag -f 
distribution/docker
 
 ## Run
 
-Edit `environment` to suite. Run 'docker-compose -f 
distribution/docker/docker-compose.yml up`
+Edit `environment` to suite. Run `docker-compose -f 
distribution/docker/docker-compose.yml up`
 
-## mysql database connector
+## MySQL Database Connector
 
 This image contains solely the postgres metadata database connector. If you 
need
 the mysql metadata storage connector, consider adding these lines before the 
`addgroup`
@@ -37,6 +37,6 @@ RUN wget -O 
/opt/druid/extensions/mysql-metadata-storage/mysql-connector-java-5.
  && ln -s 
/opt/druid/extensions/mysql-metadata-storage/mysql-connector-java-5.1.38.jar 
/opt/druid/lib
 ```
 
-Alternatively, cd src/distribution/docker; docker build -t druid:mysql 
--build-arg DRUID_RELEASE=upstream -f Dockerfile.mysql .
+Alternatively, `cd src/distribution/docker; docker build -t druid:mysql 
--build-arg DRUID_RELEASE=upstream -f Dockerfile.mysql .`
 
 where `upstream` is the version to use as the base (e.g. druid:0.14.0 from 
Dockerhub)
diff --git a/distribution/docker/druid.sh b/distribution/docker/druid.sh
index b142ec6..c60b62a 100755
--- a/distribution/docker/druid.sh
+++ b/distribution/docker/druid.sh
@@ -45,45 +45,56 @@ echo "$(date -Is) startup service $SERVICE"
 
 # We put all the config in /tmp/conf to allow for a
 # read-only root filesystem
-cp -r /opt/druid/conf /tmp/conf
+mkdir -p /tmp/conf/
+cp -r /opt/druid/conf/druid /tmp/conf/druid
+
+getConfPath() {
+cluster_conf_base=/tmp/conf/druid/cluster
+case "$1" in
+_common) echo $cluster_conf_base/_common ;;
+historical) echo $cluster_conf_base/data/historical ;;
+middleManager) echo $cluster_conf_base/data/middleManager ;;
+coordinator | overlord) echo 
$cluster_conf_base/master/coordinator-overlord ;;
+broker) echo $cluster_conf_base/query/broker ;;
+router) echo $cluster_conf_base/query/router ;;
+esac
+}
+COMMON_CONF_DIR=$(getConfPath _common)
+SERVICE_CONF_DIR=$(getConfPath ${SERVICE})
 
 # Delete the old key (if existing) and append new key=value
 setKey() {
 service="$1"
 key="$2"
 value="$3"
-case "$service" in
-_common)
-  fname=common.runtime.properties ;;
-*)
-  fname=runtime.properties ;;
-esac
+service_conf=$(getConfPath $service)/runtime.properties
 # Delete from all
-sed -ri "/$key=/d" /tmp/conf/druid/_common/common.runtime.properties
-[ -f /tmp/conf/druid/$service/$fname ] && sed -ri "/$key=/d" 
/tmp/conf/druid/$service/$fname
-[ -f /tmp/conf/druid/$service/$fname ] && echo "$key=$value" >> 
/tmp/conf/druid/$service/$fname
-[ -f /tmp/conf/druid/$service/$fname ] || echo "$key=$value" >> 
/tmp/conf/druid/_common/$fname
+sed -ri "/$key=/d" $COMMON_CONF_DIR/common.runtime.properties
+[ -f $service_conf ] && sed -ri "/$key=/d" $service_conf
+[ -f $service_conf ] && echo "$key=$value" >>$service_conf
+[ -f $service_conf ] || echo "$key=$value" 
>>$COMMON_CONF_DIR/common.runtime.pro

[incubator-druid] branch master updated: Add shuffleSegmentPusher for data shuffle (#8115)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new ab5b3be  Add shuffleSegmentPusher for data shuffle (#8115)
ab5b3be is described below

commit ab5b3be6c61d7be540f85385327a6a6489b363b1
Author: Jihoon Son 
AuthorDate: Mon Aug 5 13:38:35 2019 -0700

Add shuffleSegmentPusher for data shuffle (#8115)

* Fix race between canHandle() and addSegment() in StorageLocation

* add comment

* Add shuffleSegmentPusher which is a dataSegmentPusher used for writing 
shuffle data in local storage.

* add comments

* unused import

* add comments

* fix test

* address comments

* remove  tag from javadoc

* address comments

* comparingLong

* Address comments

* fix test
---
 .../druid/indexing/common/config/TaskConfig.java   |   5 +
 .../indexing/worker/IntermediaryDataManager.java   | 166 +
 .../indexing/worker/ShuffleDataSegmentPusher.java  |  77 ++
 .../IntermediaryDataManagerAutoCleanupTest.java|  15 +-
 ...ermediaryDataManagerManualAddAndDeleteTest.java |  47 +++---
 .../worker/ShuffleDataSegmentPusherTest.java   | 138 +
 .../druid/segment/loading/StorageLocation.java |   7 +-
 ...mentLoaderLocalCacheManagerConcurrencyTest.java |   2 -
 .../druid/segment/loading/StorageLocationTest.java |  14 +-
 9 files changed, 366 insertions(+), 105 deletions(-)

diff --git 
a/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
 
b/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
index 31405fe..52bf083 100644
--- 
a/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
+++ 
b/indexing-service/src/main/java/org/apache/druid/indexing/common/config/TaskConfig.java
@@ -126,6 +126,11 @@ public class TaskConfig
 return new File(getTaskDir(taskId), "work");
   }
 
+  public File getTaskTempDir(String taskId)
+  {
+return new File(getTaskDir(taskId), "temp");
+  }
+
   public File getTaskLockFile(String taskId)
   {
 return new File(getTaskDir(taskId), "lock");
diff --git 
a/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
 
b/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
index c47425a..95acb22 100644
--- 
a/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
+++ 
b/indexing-service/src/main/java/org/apache/druid/indexing/worker/IntermediaryDataManager.java
@@ -33,11 +33,13 @@ import org.apache.druid.java.util.common.DateTimes;
 import org.apache.druid.java.util.common.IOE;
 import org.apache.druid.java.util.common.ISE;
 import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.io.Closer;
 import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
 import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
 import org.apache.druid.java.util.common.logger.Logger;
 import org.apache.druid.segment.loading.StorageLocation;
 import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.utils.CompressionUtils;
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.Period;
@@ -45,6 +47,7 @@ import org.joda.time.Period;
 import javax.annotation.Nullable;
 import java.io.File;
 import java.io.IOException;
+import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.Arrays;
 import java.util.Collections;
@@ -57,12 +60,14 @@ import java.util.Map.Entry;
 import java.util.Set;
 import java.util.concurrent.ConcurrentHashMap;
 import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
+import java.util.stream.IntStream;
 
 /**
  * This class manages intermediary segments for data shuffle between native 
parallel index tasks.
- * In native parallel indexing, phase 1 tasks store segment files in local 
storage of middleManagers
+ * In native parallel indexing, phase 1 tasks store segment files in local 
storage of middleManagers (or indexer)
  * and phase 2 tasks read those files via HTTP.
  *
  * The directory where segment files are placed is structured as
@@ -75,11 +80,12 @@ import java.util.stream.Collectors;
 @ManageLifecycle
 public class IntermediaryDataManager
 {
-  private static final Logger log = new Logger(IntermediaryDataManager.class);
+  private static final Logger LOG = new Logger(IntermediaryDataManager.class);
 
   private final long intermediaryPartitionDiscoveryPeriodSec;
   private final long intermediaryPartitionCleanupPeriodSec;
   private final Perio

[incubator-druid] branch master updated: add noop type name to prevent jackson exception when setting type to noop (#8133)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 31a29d8  add noop type name to prevent jackson exception when setting 
type to noop (#8133)
31a29d8 is described below

commit 31a29d888315947eb5c95f304c0cf06689be89a6
Author: Parag Jain 
AuthorDate: Fri Jul 26 04:37:08 2019 +0530

add noop type name to prevent jackson exception when setting type to noop 
(#8133)
---
 .../java/org/apache/druid/server/log/NoopRequestLoggerProvider.java | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/server/src/main/java/org/apache/druid/server/log/NoopRequestLoggerProvider.java
 
b/server/src/main/java/org/apache/druid/server/log/NoopRequestLoggerProvider.java
index bff0f01..d1200f2 100644
--- 
a/server/src/main/java/org/apache/druid/server/log/NoopRequestLoggerProvider.java
+++ 
b/server/src/main/java/org/apache/druid/server/log/NoopRequestLoggerProvider.java
@@ -19,10 +19,12 @@
 
 package org.apache.druid.server.log;
 
+import com.fasterxml.jackson.annotation.JsonTypeName;
 import org.apache.druid.java.util.common.logger.Logger;
 
 /**
  */
+@JsonTypeName("noop")
 public class NoopRequestLoggerProvider implements RequestLoggerProvider
 {
   private static final Logger log = new 
Logger(NoopRequestLoggerProvider.class);


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: Spotbugs: RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN (#8076)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 714a272  Spotbugs: RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN (#8076)
714a272 is described below

commit 714a2726a7878c13e8bf5049f841ee1d34f6f83d
Author: Fokko Driesprong 
AuthorDate: Thu Jul 25 01:17:02 2019 +0200

Spotbugs: RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN (#8076)

* Spotbugs: RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN

* Use Objects instead
---
 codestyle/spotbugs-exclude.xml  | 1 -
 .../java/org/apache/druid/query/metadata/metadata/SegmentAnalysis.java  | 2 +-
 processing/src/main/java/org/apache/druid/query/scan/ScanQuery.java | 2 +-
 3 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/codestyle/spotbugs-exclude.xml b/codestyle/spotbugs-exclude.xml
index 44acdb8..eb112b2 100644
--- a/codestyle/spotbugs-exclude.xml
+++ b/codestyle/spotbugs-exclude.xml
@@ -75,7 +75,6 @@
 
 
 
-
 
 
 
diff --git 
a/processing/src/main/java/org/apache/druid/query/metadata/metadata/SegmentAnalysis.java
 
b/processing/src/main/java/org/apache/druid/query/metadata/metadata/SegmentAnalysis.java
index 726d51e..5b97cea 100644
--- 
a/processing/src/main/java/org/apache/druid/query/metadata/metadata/SegmentAnalysis.java
+++ 
b/processing/src/main/java/org/apache/druid/query/metadata/metadata/SegmentAnalysis.java
@@ -159,7 +159,7 @@ public class SegmentAnalysis implements 
Comparable
 SegmentAnalysis that = (SegmentAnalysis) o;
 return size == that.size &&
numRows == that.numRows &&
-   rollup == that.rollup &&
+   Objects.equals(rollup, that.rollup) &&
Objects.equals(id, that.id) &&
Objects.equals(interval, that.interval) &&
Objects.equals(columns, that.columns) &&
diff --git 
a/processing/src/main/java/org/apache/druid/query/scan/ScanQuery.java 
b/processing/src/main/java/org/apache/druid/query/scan/ScanQuery.java
index 3f6d407..7b314ce 100644
--- a/processing/src/main/java/org/apache/druid/query/scan/ScanQuery.java
+++ b/processing/src/main/java/org/apache/druid/query/scan/ScanQuery.java
@@ -312,7 +312,7 @@ public class ScanQuery extends BaseQuery
 final ScanQuery scanQuery = (ScanQuery) o;
 return batchSize == scanQuery.batchSize &&
limit == scanQuery.limit &&
-   legacy == scanQuery.legacy &&
+   Objects.equals(legacy, scanQuery.legacy) &&
Objects.equals(virtualColumns, scanQuery.virtualColumns) &&
Objects.equals(resultFormat, scanQuery.resultFormat) &&
Objects.equals(dimFilter, scanQuery.dimFilter) &&


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: More documentation formatting fixes (#8149)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new c87b47e  More documentation formatting fixes (#8149)
c87b47e is described below

commit c87b47e0fa5d5b93593fca43933f6389ebecef0d
Author: Magnus Henoch 
AuthorDate: Wed Jul 24 23:26:03 2019 +0100

More documentation formatting fixes (#8149)

Add empty lines before bulleted lists and code blocks, to ensure that
they show up properly on the web site.  See also #8079.
---
 .../extensions-core/approximate-histograms.md  |  1 +
 .../development/extensions-core/bloom-filter.md|  1 +
 .../extensions-core/druid-basic-security.md|  2 ++
 docs/content/ingestion/native_tasks.md |  1 +
 docs/content/operations/basic-cluster-tuning.md|  2 ++
 docs/content/operations/deep-storage-migration.md  |  1 +
 docs/content/operations/export-metadata.md | 22 --
 docs/content/operations/metadata-migration.md  |  1 +
 docs/content/operations/recommendations.md |  1 +
 docs/content/querying/aggregations.md  |  1 +
 docs/content/tutorials/cluster.md  |  9 +
 11 files changed, 32 insertions(+), 10 deletions(-)

diff --git a/docs/content/development/extensions-core/approximate-histograms.md 
b/docs/content/development/extensions-core/approximate-histograms.md
index 30b5f32..2e900d2 100644
--- a/docs/content/development/extensions-core/approximate-histograms.md
+++ b/docs/content/development/extensions-core/approximate-histograms.md
@@ -37,6 +37,7 @@ The Approximate Histogram aggregator is deprecated. Please 
use http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf](http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf)
 to compute approximate histograms, with the following modifications:
+
 - some tradeoffs in accuracy were made in the interest of speed (see below)
 - the sketch maintains the exact original data as long as the number of
   distinct data points is fewer than the resolutions (number of centroids),
diff --git a/docs/content/development/extensions-core/bloom-filter.md 
b/docs/content/development/extensions-core/bloom-filter.md
index 3d6749a..f0563a0 100644
--- a/docs/content/development/extensions-core/bloom-filter.md
+++ b/docs/content/development/extensions-core/bloom-filter.md
@@ -33,6 +33,7 @@ to use with Druid for cases where an explicit filter is 
impossible, e.g. filteri
  values.
  
 Following are some characteristics of BloomFilters:
+
 - BloomFilters are highly space efficient when compared to using a HashSet.
 - Because of the probabilistic nature of bloom filters, false positive results 
are possible (element was not actually 
 inserted into a bloom filter during construction, but `test()` says true)
diff --git a/docs/content/development/extensions-core/druid-basic-security.md 
b/docs/content/development/extensions-core/druid-basic-security.md
index e067fdf..4189c3f 100644
--- a/docs/content/development/extensions-core/druid-basic-security.md
+++ b/docs/content/development/extensions-core/druid-basic-security.md
@@ -25,6 +25,7 @@ title: "Basic Security"
 # Druid Basic Security
 
 This Apache Druid (incubating) extension adds:
+
 - an Authenticator which supports [HTTP Basic 
authentication](https://en.wikipedia.org/wiki/Basic_access_authentication)
 - an Authorizer which implements basic role-based access control
 
@@ -342,6 +343,7 @@ Unassign role {roleName} from user {userName}
 Set the permissions of {roleName}. This replaces the previous set of 
permissions on the role.
 
 Content: List of JSON Resource-Action objects, e.g.:
+
 ```
 [
 {
diff --git a/docs/content/ingestion/native_tasks.md 
b/docs/content/ingestion/native_tasks.md
index c5cd91b..1cf5e01 100644
--- a/docs/content/ingestion/native_tasks.md
+++ b/docs/content/ingestion/native_tasks.md
@@ -55,6 +55,7 @@ the implementation of splittable firehoses. Please note that 
multiple tasks can
 if one of them fails.
 
 You may want to consider the below points:
+
 - Since this task doesn't shuffle intermediate data, it isn't available for 
[perfect rollup](../ingestion/index.html#roll-up-modes).
 - The number of tasks for parallel ingestion is decided by `maxNumSubTasks` in 
the tuningConfig.
   Since the supervisor task creates up to `maxNumSubTasks` worker tasks 
regardless of the available task slots,
diff --git a/docs/content/operations/basic-cluster-tuning.md 
b/docs/content/operations/basic-cluster-tuning.md
index aa09c07..226cc92 100644
--- a/docs/content/operations/basic-cluster-tuning.md
+++ b/docs/content/operations/basic-cluster-tuning.md
@@ -37,6 +37,7 @@ If you have questions on tuning Druid for specific use cases, 
or questions on co
  Heap sizing
 
 The biggest contributions to heap usage on Historicals are:
+
 - Partial unmerged que

[incubator-druid] branch master updated: add Class.getCanonicalName to forbidden-apis (#8086)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 15fbf59  add Class.getCanonicalName to forbidden-apis (#8086)
15fbf59 is described below

commit 15fbf5983d9d6abaeda54296f97fe65f1d6bba98
Author: Clint Wylie 
AuthorDate: Tue Jul 16 15:21:50 2019 -0700

add Class.getCanonicalName to forbidden-apis (#8086)

* add checkstyle to forbid unecessary use of Class.getCanonicalName

* use forbiddin-api instead of checkstyle

* add space
---
 codestyle/druid-forbidden-apis.txt   |  3 ++-
 .../java/org/apache/druid/guice/GuiceInjectableValues.java   |  2 +-
 .../apache/druid/java/util/common/lifecycle/Lifecycle.java   |  6 ++
 .../org/apache/druid/metadata/DefaultPasswordProvider.java   |  2 +-
 .../druid/query/lookup/NamespaceLookupExtractorFactory.java  |  2 +-
 .../cache/OffHeapNamespaceExtractionCacheManager.java|  2 +-
 .../query/lookup/namespace/UriExtractionNamespaceTest.java   |  8 
 .../druid/indexing/overlord/http/OverlordResource.java   |  2 +-
 .../seekablestream/SeekableStreamDataSourceMetadata.java |  8 
 .../seekablestream/SeekableStreamEndSequenceNumbers.java |  8 
 .../seekablestream/SeekableStreamStartSequenceNumbers.java   |  8 
 .../apache/druid/indexing/worker/http/WorkerResource.java|  2 +-
 .../apache/druid/collections/bitmap/WrappedBitSetBitmap.java | 12 ++--
 .../druid/query/aggregation/first/StringFirstAggregator.java |  2 +-
 .../query/aggregation/first/StringFirstBufferAggregator.java |  2 +-
 .../druid/query/aggregation/last/StringLastAggregator.java   |  2 +-
 .../query/aggregation/last/StringLastBufferAggregator.java   |  2 +-
 .../DataSourceMetadataQueryRunnerFactory.java|  2 +-
 .../druid/query/lookup/RegisteredLookupExtractionFn.java |  2 +-
 .../java/org/apache/druid/query/search/SearchStrategy.java   |  2 +-
 .../org/apache/druid/granularity/QueryGranularityTest.java   |  2 +-
 .../test/java/org/apache/druid/segment/CloserRuleTest.java   |  2 +-
 .../apache/druid/segment/data/BitmapCreationBenchmark.java   |  2 +-
 .../java/org/apache/druid/initialization/Initialization.java |  4 ++--
 .../druid/initialization/Log4jShutterDownerModule.java   |  8 
 .../main/java/org/apache/druid/server/StatusResource.java|  3 +--
 .../apache/druid/server/coordinator/DruidCoordinator.java|  2 +-
 .../org/apache/druid/initialization/InitializationTest.java  |  5 ++---
 .../java/org/apache/druid/server/StatusResourceTest.java |  2 +-
 29 files changed, 53 insertions(+), 56 deletions(-)

diff --git a/codestyle/druid-forbidden-apis.txt 
b/codestyle/druid-forbidden-apis.txt
index b06e502..c7f38fb 100644
--- a/codestyle/druid-forbidden-apis.txt
+++ b/codestyle/druid-forbidden-apis.txt
@@ -34,6 +34,7 @@ java.util.Random#() @ Use ThreadLocalRandom.current() 
or the constructor w
 java.lang.Math#random() @ Use ThreadLocalRandom.current()
 java.util.regex.Pattern#matches(java.lang.String,java.lang.CharSequence) @ Use 
String.startsWith(), endsWith(), contains(), or compile and cache a Pattern 
explicitly
 org.apache.commons.io.FileUtils#getTempDirectory() @ Use 
org.junit.rules.TemporaryFolder for tests instead
+java.lang.Class#getCanonicalName() @ Class.getCanonicalName can return null 
for anonymous types, use Class.getName instead.
 
 @defaultMessage Use Locale.ENGLISH
 com.ibm.icu.text.DateFormatSymbols#()
@@ -45,4 +46,4 @@ org.apache.commons.codec.binary.Base64
 com.google.common.io.BaseEncoding.base64
 
 @defaultMessage Use com.google.errorprone.annotations.concurrent.GuardedBy
-javax.annotations.concurrent.GuardedBy
\ No newline at end of file
+javax.annotations.concurrent.GuardedBy
diff --git 
a/core/src/main/java/org/apache/druid/guice/GuiceInjectableValues.java 
b/core/src/main/java/org/apache/druid/guice/GuiceInjectableValues.java
index f7071c9..d12497e 100644
--- a/core/src/main/java/org/apache/druid/guice/GuiceInjectableValues.java
+++ b/core/src/main/java/org/apache/druid/guice/GuiceInjectableValues.java
@@ -54,7 +54,7 @@ public class GuiceInjectableValues extends InjectableValues
 }
 throw new IAE(
 "Unknown class type [%s] for valueId [%s]",
-valueId.getClass().getCanonicalName(),
+valueId.getClass().getName(),
 valueId.toString()
 );
   }
diff --git 
a/core/src/main/java/org/apache/druid/java/util/common/lifecycle/Lifecycle.java 
b/core/src/main/java/org/apache/druid/java/util/common/lifecycle/Lifecycle.java
index 2cc897b..413f4ba 100644
--- 
a/core/src/main/java/org/apache/druid/java/util/common/lifecycle/Lifecycle.java
+++ 
b/core/src/main/java/org/apache/druid/java/util/common/lifecycle/Lifecycle.java
@@ -432,8 +432,7 @@ public class Lifecycle
   for (Met

[incubator-druid] branch master updated: fail complex type 'serde' registration when registered type does not match expected type (#7985)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new abf9843  fail complex type 'serde' registration when registered type 
does not match expected type (#7985)
abf9843 is described below

commit abf9843e2a217656c4fa4bc79ffd771b13680281
Author: Clint Wylie 
AuthorDate: Thu Jul 11 23:03:16 2019 -0700

fail complex type 'serde' registration when registered type does not match 
expected type (#7985)

* make ComplexMetrics.registerSerde type check on register, resolves #7959

* add test

* simplify

* unused imports :/

* simplify

* burned by imports yet again

* remove unused constructor

* switch to getName

* heh oops
---
 .../druid/benchmark/FilterPartitionBenchmark.java  |  3 +-
 .../benchmark/FilteredAggregatorBenchmark.java |  3 +-
 .../benchmark/GroupByTypeInterfaceBenchmark.java   |  3 +-
 .../benchmark/TopNTypeInterfaceBenchmark.java  |  3 +-
 .../druid/benchmark/datagen/SegmentGenerator.java  |  3 +-
 .../indexing/IncrementalIndexReadBenchmark.java|  3 +-
 .../indexing/IndexIngestionBenchmark.java  |  3 +-
 .../benchmark/indexing/IndexMergeBenchmark.java|  3 +-
 .../benchmark/indexing/IndexPersistBenchmark.java  |  3 +-
 .../druid/benchmark/query/GroupByBenchmark.java|  3 +-
 .../druid/benchmark/query/ScanBenchmark.java   |  3 +-
 .../druid/benchmark/query/SearchBenchmark.java |  3 +-
 .../druid/benchmark/query/SelectBenchmark.java |  3 +-
 .../druid/benchmark/query/TimeseriesBenchmark.java |  3 +-
 .../druid/benchmark/query/TopNBenchmark.java   |  3 +-
 .../query/timecompare/TimeCompareBenchmark.java|  3 +-
 .../momentsketch/MomentSketchModule.java   |  2 +-
 .../tdigestsketch/TDigestSketchModule.java |  3 +-
 .../datasketches/hll/HllSketchModule.java  |  6 +-
 .../quantiles/DoublesSketchModule.java |  2 +-
 .../datasketches/theta/SketchModule.java   |  6 +-
 .../theta/oldapi/OldApiSketchModule.java   | 12 ++--
 .../tuple/ArrayOfDoublesSketchModule.java  |  6 +-
 .../druid/guice/BloomFilterSerializersModule.java  |  2 +-
 .../histogram/ApproximateHistogramDruidModule.java |  4 +-
 .../query/aggregation/stats/DruidStatsModule.java  |  2 +-
 .../apache/druid/jackson/AggregatorsModule.java| 10 +--
 .../aggregation/hyperloglog/HyperUniquesSerde.java |  5 ++
 .../hyperloglog/PreComputedHyperUniquesSerde.java  |  5 +-
 .../apache/druid/segment/serde/ComplexMetrics.java | 17 +++--
 .../apache/druid/segment/SchemalessIndexTest.java  |  3 +-
 .../java/org/apache/druid/segment/TestIndex.java   |  3 +-
 .../druid/segment/serde/ComplexMetricsTest.java| 78 ++
 33 files changed, 139 insertions(+), 75 deletions(-)

diff --git 
a/benchmarks/src/main/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
 
b/benchmarks/src/main/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
index ecf5027..0309ac7 100644
--- 
a/benchmarks/src/main/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
+++ 
b/benchmarks/src/main/java/org/apache/druid/benchmark/FilterPartitionBenchmark.java
@@ -29,7 +29,6 @@ import org.apache.druid.benchmark.datagen.BenchmarkSchemaInfo;
 import org.apache.druid.benchmark.datagen.BenchmarkSchemas;
 import org.apache.druid.common.config.NullHandling;
 import org.apache.druid.data.input.InputRow;
-import org.apache.druid.hll.HyperLogLogHash;
 import org.apache.druid.jackson.DefaultObjectMapper;
 import org.apache.druid.java.util.common.granularity.Granularities;
 import org.apache.druid.java.util.common.guava.Sequence;
@@ -149,7 +148,7 @@ public class FilterPartitionBenchmark
   {
 log.info("SETUP CALLED AT " + System.currentTimeMillis());
 
-ComplexMetrics.registerSerde("hyperUnique", () -> new 
HyperUniquesSerde(HyperLogLogHash.getDefault()));
+ComplexMetrics.registerSerde("hyperUnique", new HyperUniquesSerde());
 
 schemaInfo = BenchmarkSchemas.SCHEMA_MAP.get(schema);
 
diff --git 
a/benchmarks/src/main/java/org/apache/druid/benchmark/FilteredAggregatorBenchmark.java
 
b/benchmarks/src/main/java/org/apache/druid/benchmark/FilteredAggregatorBenchmark.java
index 4aafcc3..448cc00 100644
--- 
a/benchmarks/src/main/java/org/apache/druid/benchmark/FilteredAggregatorBenchmark.java
+++ 
b/benchmarks/src/main/java/org/apache/druid/benchmark/FilteredAggregatorBenchmark.java
@@ -27,7 +27,6 @@ import org.apache.druid.benchmark.datagen.BenchmarkSchemaInfo;
 import org.apache.druid.benchmark.datagen.BenchmarkSchemas;
 import org.apache.druid.benchmark.query.QueryBenchmarkUtil;
 import org.apache.druid.data.input.InputRow;
-import org.apache.druid.hll.HyperLogLogHash;
 import org.apache.druid.jac

[incubator-druid] branch master updated: add config to optionally disable all compression in intermediate segment persists while ingestion (#7919)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 14aec7f  add config to optionally disable all compression  in 
intermediate segment persists while ingestion (#7919)
14aec7f is described below

commit 14aec7fceca90dfaf9b2ce4dae68186d04ffcc47
Author: Himanshu 
AuthorDate: Wed Jul 10 12:22:24 2019 -0700

add config to optionally disable all compression  in intermediate segment 
persists while ingestion (#7919)

* disable all compression in intermediate segment persists while ingestion

* more changes and build fix

* by default retain existing indexingSpec for intermediate persisted 
segments

* document indexSpecForIntermediatePersists index tuning config

* fix build issues

* update serde tests
---
 .../development/extensions-core/kafka-ingestion.md |  3 +-
 .../extensions-core/kinesis-ingestion.md   |  3 +-
 docs/content/ingestion/hadoop.md   |  3 +-
 docs/content/ingestion/native_tasks.md | 10 ++
 .../MaterializedViewSupervisorSpec.java|  1 +
 .../indexing/kafka/KafkaIndexTaskTuningConfig.java |  4 +++
 .../kafka/supervisor/KafkaSupervisorSpec.java  |  1 +
 .../supervisor/KafkaSupervisorTuningConfig.java|  3 ++
 .../druid/indexing/kafka/KafkaIndexTaskTest.java   |  1 +
 .../kafka/KafkaIndexTaskTuningConfigTest.java  | 12 ++-
 .../kafka/supervisor/KafkaSupervisorTest.java  |  2 ++
 .../KafkaSupervisorTuningConfigTest.java   |  8 -
 .../TestModifiedKafkaIndexTaskTuningConfig.java|  2 ++
 .../kinesis/KinesisIndexTaskTuningConfig.java  |  3 ++
 .../kinesis/supervisor/KinesisSupervisorSpec.java  |  1 +
 .../supervisor/KinesisSupervisorTuningConfig.java  |  3 ++
 .../indexing/kinesis/KinesisIndexTaskTest.java |  1 +
 .../kinesis/KinesisIndexTaskTuningConfigTest.java  |  3 ++
 .../kinesis/supervisor/KinesisSupervisorTest.java  |  2 ++
 .../TestModifiedKinesisIndexTaskTuningConfig.java  |  3 ++
 .../druid/indexer/HadoopDruidIndexerConfig.java|  5 +++
 .../apache/druid/indexer/HadoopTuningConfig.java   | 14 
 .../apache/druid/indexer/IndexGeneratorJob.java|  2 +-
 .../druid/indexer/BatchDeltaIngestionTest.java |  1 +
 .../indexer/DetermineHashedPartitionsJobTest.java  |  1 +
 .../druid/indexer/DeterminePartitionsJobTest.java  |  1 +
 .../indexer/HadoopDruidIndexerConfigTest.java  |  2 ++
 .../druid/indexer/HadoopTuningConfigTest.java  |  2 ++
 .../druid/indexer/IndexGeneratorJobTest.java   |  1 +
 .../org/apache/druid/indexer/JobHelperTest.java|  1 +
 .../indexer/path/GranularityPathSpecTest.java  |  1 +
 .../index/RealtimeAppenderatorTuningConfig.java| 12 +++
 .../indexing/common/index/YeOldePlumberSchool.java |  2 +-
 .../druid/indexing/common/task/IndexTask.java  | 40 +-
 .../parallel/ParallelIndexSupervisorTask.java  |  1 +
 .../batch/parallel/ParallelIndexTuningConfig.java  |  4 ++-
 .../SeekableStreamIndexTaskTuningConfig.java   | 13 +++
 .../AppenderatorDriverRealtimeIndexTaskTest.java   |  1 +
 .../indexing/common/task/CompactionTaskTest.java   |  6 
 .../druid/indexing/common/task/IndexTaskTest.java  |  4 +++
 .../common/task/RealtimeIndexTaskTest.java |  1 +
 .../druid/indexing/common/task/TaskSerdeTest.java  |  3 ++
 .../ParallelIndexSupervisorTaskKillTest.java   |  1 +
 .../ParallelIndexSupervisorTaskResourceTest.java   |  1 +
 .../ParallelIndexSupervisorTaskSerdeTest.java  |  1 +
 .../parallel/ParallelIndexSupervisorTaskTest.java  |  2 ++
 .../parallel/ParallelIndexTuningConfigTest.java|  1 +
 .../druid/indexing/overlord/TaskLifecycleTest.java |  4 +++
 .../SeekableStreamSupervisorStateTest.java |  1 +
 .../segment/indexing/RealtimeTuningConfig.java | 14 
 .../realtime/appenderator/AppenderatorConfig.java  |  2 ++
 .../realtime/appenderator/AppenderatorImpl.java|  4 +--
 .../segment/realtime/plumber/RealtimePlumber.java  |  5 +--
 .../segment/indexing/RealtimeTuningConfigTest.java | 10 --
 .../appenderator/AppenderatorPlumberTest.java  |  1 +
 .../realtime/appenderator/AppenderatorTester.java  |  1 +
 .../DefaultOfflineAppenderatorFactoryTest.java |  1 +
 .../plumber/RealtimePlumberSchoolTest.java |  1 +
 .../druid/segment/realtime/plumber/SinkTest.java   |  2 ++
 .../druid/cli/validate/DruidJsonValidatorTest.java |  1 +
 60 files changed, 215 insertions(+), 25 deletions(-)

diff --git a/docs/content/development/extensions-core/kafka-ingestion.md 
b/docs/content/development/extensions-core/kafka-ingestion.md
index c070e46..ec1d046 100644
--- a/docs/content/development/extensions-core/kafka-ingestion.md
+++ b/docs/content/development/extensions-core/kafka-ingestion.md
@@ -139,7 +139,8

[incubator-druid] branch master updated: Fix HttpRemoteTaskRunnerConfig not loading (#7902)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new daed1bc  Fix HttpRemoteTaskRunnerConfig not loading (#7902)
daed1bc is described below

commit daed1bc1417e832109ea5177d2724e0c8576e6ce
Author: litao91 
AuthorDate: Sun Jun 16 15:31:42 2019 +0800

Fix HttpRemoteTaskRunnerConfig not loading (#7902)
---
 .../main/java/org/apache/druid/guice/IndexingServiceModuleHelper.java   | 2 ++
 1 file changed, 2 insertions(+)

diff --git 
a/indexing-service/src/main/java/org/apache/druid/guice/IndexingServiceModuleHelper.java
 
b/indexing-service/src/main/java/org/apache/druid/guice/IndexingServiceModuleHelper.java
index 5135c97..cc37324 100644
--- 
a/indexing-service/src/main/java/org/apache/druid/guice/IndexingServiceModuleHelper.java
+++ 
b/indexing-service/src/main/java/org/apache/druid/guice/IndexingServiceModuleHelper.java
@@ -21,6 +21,7 @@ package org.apache.druid.guice;
 
 import com.google.inject.Binder;
 import org.apache.druid.indexing.overlord.config.ForkingTaskRunnerConfig;
+import org.apache.druid.indexing.overlord.config.HttpRemoteTaskRunnerConfig;
 import org.apache.druid.indexing.overlord.config.RemoteTaskRunnerConfig;
 import org.apache.druid.server.initialization.IndexerZkConfig;
 
@@ -34,6 +35,7 @@ public class IndexingServiceModuleHelper
   {
 JsonConfigProvider.bind(binder, INDEXER_RUNNER_PROPERTY_PREFIX, 
ForkingTaskRunnerConfig.class);
 JsonConfigProvider.bind(binder, INDEXER_RUNNER_PROPERTY_PREFIX, 
RemoteTaskRunnerConfig.class);
+JsonConfigProvider.bind(binder, INDEXER_RUNNER_PROPERTY_PREFIX, 
HttpRemoteTaskRunnerConfig.class);
 JsonConfigProvider.bind(binder, "druid.zk.paths.indexer", 
IndexerZkConfig.class);
   }
 }


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: endpoint to delete lookup tier and remove tier on last lookup deletion (#7852)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new b3328b2  endpoint to delete lookup tier and remove tier on last lookup 
deletion (#7852)
b3328b2 is described below

commit b3328b2785884a4677338d97a116ef1fdb54459f
Author: Himanshu 
AuthorDate: Sat Jun 15 17:55:50 2019 -0700

endpoint to delete lookup tier and remove tier on last lookup deletion 
(#7852)
---
 docs/content/querying/lookups.md   |  5 +-
 .../server/http/LookupCoordinatorResource.java | 29 
 .../lookup/cache/LookupCoordinatorManager.java | 28 ++-
 .../server/http/LookupCoordinatorResourceTest.java | 42 +++
 .../lookup/cache/LookupCoordinatorManagerTest.java | 87 ++
 5 files changed, 189 insertions(+), 2 deletions(-)

diff --git a/docs/content/querying/lookups.md b/docs/content/querying/lookups.md
index b54f769..7af2bcd 100644
--- a/docs/content/querying/lookups.md
+++ b/docs/content/querying/lookups.md
@@ -292,7 +292,10 @@ Using the prior example, a `GET` to 
`/druid/coordinator/v1/lookups/config/realti
 ```
 
 ## Delete Lookup
-A `DELETE` to `/druid/coordinator/v1/lookups/config/{tier}/{id}` will remove 
that lookup from the cluster.
+A `DELETE` to `/druid/coordinator/v1/lookups/config/{tier}/{id}` will remove 
that lookup from the cluster. If it was last lookup in the tier, then tier is 
deleted as well.
+
+## Delete Tier
+A `DELETE` to `/druid/coordinator/v1/lookups/config/{tier}` will remove that 
tier from the cluster.
 
 ## List tier names
 A `GET` to `/druid/coordinator/v1/lookups/config` will return a list of known 
tier names in the dynamic configuration.
diff --git 
a/server/src/main/java/org/apache/druid/server/http/LookupCoordinatorResource.java
 
b/server/src/main/java/org/apache/druid/server/http/LookupCoordinatorResource.java
index 7affefe..f575211 100644
--- 
a/server/src/main/java/org/apache/druid/server/http/LookupCoordinatorResource.java
+++ 
b/server/src/main/java/org/apache/druid/server/http/LookupCoordinatorResource.java
@@ -178,6 +178,35 @@ public class LookupCoordinatorResource
 
   @DELETE
   @Produces({MediaType.APPLICATION_JSON, 
SmileMediaTypes.APPLICATION_JACKSON_SMILE})
+  @Path("/config/{tier}")
+  public Response deleteTier(
+  @PathParam("tier") String tier,
+  @HeaderParam(AuditManager.X_DRUID_AUTHOR) @DefaultValue("") final String 
author,
+  @HeaderParam(AuditManager.X_DRUID_COMMENT) @DefaultValue("") final 
String comment,
+  @Context HttpServletRequest req
+  )
+  {
+try {
+  if (Strings.isNullOrEmpty(tier)) {
+return Response.status(Response.Status.BAD_REQUEST)
+   .entity(ServletResourceUtils.sanitizeException(new 
NullPointerException("`tier` required")))
+   .build();
+  }
+
+  if (lookupCoordinatorManager.deleteTier(tier, new AuditInfo(author, 
comment, req.getRemoteAddr( {
+return Response.status(Response.Status.ACCEPTED).build();
+  } else {
+return Response.status(Response.Status.NOT_FOUND).build();
+  }
+}
+catch (Exception e) {
+  LOG.error(e, "Error deleting tier [%s]", tier);
+  return 
Response.serverError().entity(ServletResourceUtils.sanitizeException(e)).build();
+}
+  }
+
+  @DELETE
+  @Produces({MediaType.APPLICATION_JSON, 
SmileMediaTypes.APPLICATION_JACKSON_SMILE})
   @Path("/config/{tier}/{lookup}")
   public Response deleteLookup(
   @PathParam("tier") String tier,
diff --git 
a/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
 
b/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
index 86acfa3..b6589aa 100644
--- 
a/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
+++ 
b/server/src/main/java/org/apache/druid/server/lookup/cache/LookupCoordinatorManager.java
@@ -255,6 +255,27 @@ public class LookupCoordinatorManager
 return lookupMapConfigRef.get();
   }
 
+  public boolean deleteTier(final String tier, AuditInfo auditInfo)
+  {
+Preconditions.checkState(lifecycleLock.awaitStarted(5, TimeUnit.SECONDS), 
"not started");
+
+synchronized (this) {
+  final Map> 
priorSpec = getKnownLookups();
+  if (priorSpec == null) {
+LOG.warn("Requested delete tier [%s]. But no lookups exist!", tier);
+return false;
+  }
+  final Map> 
updateSpec = new HashMap<>(priorSpec);
+
+  if (updateSpec.remove(tier) == null) {
+LOG.warn("Requested delete of tier [%s] that does not exist!", tier);
+return false;
+  }
+
+  return configManager.set(LOOKUP_CONFIG_KEY, updateSpec, 
au

[incubator-druid] branch master updated: discard filter when processing subtotalsSpec (#7827)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 0493780  discard filter when processing subtotalsSpec (#7827)
0493780 is described below

commit 04937807992417103fb612d9faa059281e11d582
Author: Himanshu 
AuthorDate: Tue Jun 4 10:59:22 2019 -0700

discard filter when processing subtotalsSpec (#7827)
---
 .../query/groupby/strategy/GroupByStrategyV2.java  |  2 +-
 .../query/groupby/GroupByQueryRunnerTest.java  | 90 ++
 2 files changed, 91 insertions(+), 1 deletion(-)

diff --git 
a/processing/src/main/java/org/apache/druid/query/groupby/strategy/GroupByStrategyV2.java
 
b/processing/src/main/java/org/apache/druid/query/groupby/strategy/GroupByStrategyV2.java
index 7a5f6e8..cd69883 100644
--- 
a/processing/src/main/java/org/apache/druid/query/groupby/strategy/GroupByStrategyV2.java
+++ 
b/processing/src/main/java/org/apache/druid/query/groupby/strategy/GroupByStrategyV2.java
@@ -390,7 +390,7 @@ public class GroupByStrategyV2 implements GroupByStrategy
 final List closeOnExit = new ArrayList<>();
 
 try {
-  GroupByQuery queryWithoutSubtotalsSpec = query.withSubtotalsSpec(null);
+  GroupByQuery queryWithoutSubtotalsSpec = 
query.withSubtotalsSpec(null).withDimFilter(null);
   List> subtotals = query.getSubtotalsSpec();
 
   Supplier grouperSupplier = Suppliers.memoize(
diff --git 
a/processing/src/test/java/org/apache/druid/query/groupby/GroupByQueryRunnerTest.java
 
b/processing/src/test/java/org/apache/druid/query/groupby/GroupByQueryRunnerTest.java
index 36b7994..27d0da9 100644
--- 
a/processing/src/test/java/org/apache/druid/query/groupby/GroupByQueryRunnerTest.java
+++ 
b/processing/src/test/java/org/apache/druid/query/groupby/GroupByQueryRunnerTest.java
@@ -6787,6 +6787,96 @@ public class GroupByQueryRunnerTest
 TestHelper.assertExpectedObjects(expectedResults, results, "subtotal");
   }
 
+  // https://github.com/apache/incubator-druid/issues/7820
+  @Test
+  public void testGroupByWithSubtotalsSpecWithRenamedDimensionAndFilter()
+  {
+if 
(!config.getDefaultStrategy().equals(GroupByStrategySelector.STRATEGY_V2)) {
+  return;
+}
+
+GroupByQuery query = GroupByQuery
+.builder()
+.setDataSource(QueryRunnerTestHelper.dataSource)
+.setQuerySegmentSpec(QueryRunnerTestHelper.firstToThird)
+.setVirtualColumns(new ExpressionVirtualColumn("alias", "quality", 
ValueType.STRING, TestExprMacroTable.INSTANCE))
+.setDimensions(Lists.newArrayList(
+new DefaultDimensionSpec("quality", "quality"),
+new DefaultDimensionSpec("market", "market"),
+new DefaultDimensionSpec("alias", "alias_renamed")
+))
+.setAggregatorSpecs(
+Arrays.asList(
+QueryRunnerTestHelper.rowsCount,
+new LongSumAggregatorFactory("idx", "index"),
+new FloatSumAggregatorFactory("idxFloat", "indexFloat"),
+new DoubleSumAggregatorFactory("idxDouble", "index")
+)
+)
+.setDimFilter(new SelectorDimFilter("alias", "automotive", null))
+.setGranularity(QueryRunnerTestHelper.dayGran)
+.setSubtotalsSpec(ImmutableList.of(
+ImmutableList.of("alias_renamed"),
+ImmutableList.of()
+))
+.build();
+
+List expectedResults = Arrays.asList(
+GroupByQueryRunnerTestHelper.createExpectedRow(
+"2011-04-01",
+"alias_renamed",
+"automotive",
+"rows",
+1L,
+"idx",
+135L,
+"idxFloat",
+135.88510131835938f,
+"idxDouble",
+135.88510131835938d
+),
+GroupByQueryRunnerTestHelper.createExpectedRow(
+"2011-04-02",
+"alias_renamed",
+"automotive",
+"rows",
+1L,
+"idx",
+147L,
+"idxFloat",
+147.42593f,
+"idxDouble",
+147.42593d
+),
+
+GroupByQueryRunnerTestHelper.createExpectedRow(
+"2011-04-01T00:00:00.000Z",
+"rows",
+1L,
+"idx",
+135L,
+"idxFloat",
+135.88510131835938f,
+"idxDouble",
+135.88510131835938d
+),
+GroupByQueryRunnerTestHelper.createExpectedRow(
+

[incubator-druid] branch master updated: Fix memory problem (OOM/FGC) when expression is used in metricsSpec (#7716)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 42cf078  Fix memory problem (OOM/FGC) when expression is used in 
metricsSpec (#7716)
42cf078 is described below

commit 42cf07884345fa5fba70a17156678dc6c37ab3d1
Author: BIGrey 
AuthorDate: Tue May 28 00:46:17 2019 +0800

Fix memory problem (OOM/FGC) when expression is used in metricsSpec (#7716)

* AggregatorUtil should cache parsed expression to avoid memory problem 
(OOM/FGC) when Expression is used in metricsSpec

* remove debug log check in Parser.parse

* remove cache and use suppliers.memorize
---
 .../druid/benchmark/datagen/BenchmarkSchemas.java  | 18 +
 .../druid/query/aggregation/AggregatorUtil.java| 20 ++---
 .../aggregation/SimpleDoubleAggregatorFactory.java | 10 ++-
 .../aggregation/SimpleFloatAggregatorFactory.java  | 10 ++-
 .../aggregation/SimpleLongAggregatorFactory.java   | 10 ++-
 .../incremental/OnheapIncrementalIndexTest.java| 94 ++
 6 files changed, 139 insertions(+), 23 deletions(-)

diff --git 
a/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
 
b/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
index cda9f47..3e35a66 100644
--- 
a/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
+++ 
b/benchmarks/src/main/java/org/apache/druid/benchmark/datagen/BenchmarkSchemas.java
@@ -21,6 +21,7 @@ package org.apache.druid.benchmark.datagen;
 
 import com.google.common.collect.ImmutableList;
 import org.apache.druid.java.util.common.Intervals;
+import org.apache.druid.math.expr.ExprMacroTable;
 import org.apache.druid.query.aggregation.AggregatorFactory;
 import org.apache.druid.query.aggregation.CountAggregatorFactory;
 import org.apache.druid.query.aggregation.DoubleMinAggregatorFactory;
@@ -85,6 +86,14 @@ public class BenchmarkSchemas
 basicSchemaIngestAggs.add(new DoubleMinAggregatorFactory("minFloatZipf", 
"metFloatZipf"));
 basicSchemaIngestAggs.add(new HyperUniquesAggregatorFactory("hyper", 
"dimHyperUnique"));
 
+List basicSchemaIngestAggsExpression = new 
ArrayList<>();
+basicSchemaIngestAggsExpression.add(new CountAggregatorFactory("rows"));
+basicSchemaIngestAggsExpression.add(new 
LongSumAggregatorFactory("sumLongSequential", null, "if(sumLongSequential>0 && 
dimSequential>100 || dimSequential<10 || 
metLongSequential>3000,sumLongSequential,0)", ExprMacroTable.nil()));
+basicSchemaIngestAggsExpression.add(new 
LongMaxAggregatorFactory("maxLongUniform", "metLongUniform"));
+basicSchemaIngestAggsExpression.add(new 
DoubleSumAggregatorFactory("sumFloatNormal", null, "if(sumFloatNormal>0 && 
dimSequential>100 || dimSequential<10 || 
metLongSequential>3000,sumFloatNormal,0)", ExprMacroTable.nil()));
+basicSchemaIngestAggsExpression.add(new 
DoubleMinAggregatorFactory("minFloatZipf", "metFloatZipf"));
+basicSchemaIngestAggsExpression.add(new 
HyperUniquesAggregatorFactory("hyper", "dimHyperUnique"));
+
 Interval basicSchemaDataInterval = Intervals.utc(0, 100);
 
 BenchmarkSchemaInfo basicSchema = new BenchmarkSchemaInfo(
@@ -93,7 +102,16 @@ public class BenchmarkSchemas
 basicSchemaDataInterval,
 true
 );
+
+BenchmarkSchemaInfo basicSchemaExpression = new BenchmarkSchemaInfo(
+basicSchemaColumns,
+basicSchemaIngestAggsExpression,
+basicSchemaDataInterval,
+true
+);
+
 SCHEMA_MAP.put("basic", basicSchema);
+SCHEMA_MAP.put("expression", basicSchemaExpression);
   }
 
   static { // simple single string column and count agg schema, no rollup
diff --git 
a/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
 
b/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
index bf1edd2..e5e5f51 100644
--- 
a/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
+++ 
b/processing/src/main/java/org/apache/druid/query/aggregation/AggregatorUtil.java
@@ -24,8 +24,6 @@ import org.apache.druid.guice.annotations.PublicApi;
 import org.apache.druid.java.util.common.Pair;
 import org.apache.druid.math.expr.Expr;
 import org.apache.druid.math.expr.ExprEval;
-import org.apache.druid.math.expr.ExprMacroTable;
-import org.apache.druid.math.expr.Parser;
 import org.apache.druid.query.monomorphicprocessing.RuntimeShapeInspector;
 import org.apache.druid.segment.BaseDoubleColumnValueSelector;
 import org.apache.druid.segment.BaseFloatColumnValueSelector;
@@ -184,9 +182,8 @@ public class Aggre

[incubator-druid] branch master updated: allow quantiles merge aggregator to also accept doubles (#7718)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 23e96d1  allow quantiles merge aggregator to also accept doubles 
(#7718)
23e96d1 is described below

commit 23e96d15d41fef0cfe4d1815421b19fca3005334
Author: Clint Wylie 
AuthorDate: Thu May 23 11:13:41 2019 -0700

allow quantiles merge aggregator to also accept doubles (#7718)

* allow quantiles merge aggregator to also accept doubles

* consolidate dupe

* import
---
 .../quantiles/DoublesSketchBuildAggregator.java|   3 -
 .../quantiles/DoublesSketchMergeAggregator.java|  23 +++--
 .../DoublesSketchMergeBufferAggregator.java|  11 +--
 .../sql/DoublesSketchSqlAggregatorTest.java| 106 +
 4 files changed, 106 insertions(+), 37 deletions(-)

diff --git 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
index bd46fc5..18f94a9 100644
--- 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
+++ 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchBuildAggregator.java
@@ -28,14 +28,12 @@ public class DoublesSketchBuildAggregator implements 
Aggregator
 {
 
   private final ColumnValueSelector valueSelector;
-  private final int size;
 
   private UpdateDoublesSketch sketch;
 
   public DoublesSketchBuildAggregator(final ColumnValueSelector 
valueSelector, final int size)
   {
 this.valueSelector = valueSelector;
-this.size = size;
 sketch = DoublesSketch.builder().setK(size).build();
   }
 
@@ -68,5 +66,4 @@ public class DoublesSketchBuildAggregator implements 
Aggregator
   {
 sketch = null;
   }
-
 }
diff --git 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
index 325a6f2..4598048 100644
--- 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
+++ 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeAggregator.java
@@ -27,10 +27,10 @@ import org.apache.druid.segment.ColumnValueSelector;
 public class DoublesSketchMergeAggregator implements Aggregator
 {
 
-  private final ColumnValueSelector selector;
+  private final ColumnValueSelector selector;
   private DoublesUnion union;
 
-  public DoublesSketchMergeAggregator(final ColumnValueSelector 
selector, final int k)
+  public DoublesSketchMergeAggregator(final ColumnValueSelector selector, 
final int k)
   {
 this.selector = selector;
 union = DoublesUnion.builder().setMaxK(k).build();
@@ -39,13 +39,10 @@ public class DoublesSketchMergeAggregator implements 
Aggregator
   @Override
   public synchronized void aggregate()
   {
-final DoublesSketch sketch = selector.getObject();
-if (sketch == null) {
-  return;
-}
-union.update(sketch);
+updateUnion(selector, union);
   }
 
+
   @Override
   public synchronized Object get()
   {
@@ -70,4 +67,16 @@ public class DoublesSketchMergeAggregator implements 
Aggregator
 union = null;
   }
 
+  static void updateUnion(ColumnValueSelector selector, DoublesUnion union)
+  {
+final Object object = selector.getObject();
+if (object == null) {
+  return;
+}
+if (object instanceof DoublesSketch) {
+  union.update((DoublesSketch) object);
+} else {
+  union.update(selector.getDouble());
+}
+  }
 }
diff --git 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
index ffe9009..f5a1e9d 100644
--- 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
+++ 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/quantiles/DoublesSketchMergeBufferAggregator.java
@@ -20,7 +20,6 @@
 package org.apache.druid.query.aggregation.datasketches.quantiles;
 
 import com.yahoo.memory.WritableMemory;
-import com.yahoo.sketches.quantiles.DoublesSketch;
 import com.yahoo.sketches.quantiles.DoublesUnion;
 import

[incubator-druid] branch master updated: Adding influxdb emitter as a contrib extension (#7717)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 6964ac2  Adding influxdb emitter as a contrib extension (#7717)
6964ac2 is described below

commit 6964ac23a2a3b95f29a2d4aa443ef9fcbc84dfdb
Author: awelsh93 <32643586+awels...@users.noreply.github.com>
AuthorDate: Thu May 23 19:11:48 2019 +0100

Adding influxdb emitter as a contrib extension (#7717)

* Adding influxdb emitter as a contrib extension

* addressing code review comments
---
 distribution/pom.xml   |   2 +
 .../extensions-contrib/influxdb-emitter.md |  75 
 docs/content/development/extensions.md |   1 +
 extensions-contrib/influxdb-emitter/pom.xml|  74 +++
 .../druid/emitter/influxdb/InfluxdbEmitter.java| 214 +
 .../emitter/influxdb/InfluxdbEmitterConfig.java| 196 +++
 .../emitter/influxdb/InfluxdbEmitterModule.java|  61 ++
 .../org.apache.druid.initialization.DruidModule|  16 ++
 .../influxdb/InfluxdbEmitterConfigTest.java| 212 
 .../emitter/influxdb/InfluxdbEmitterTest.java  | 208 
 pom.xml|   1 +
 11 files changed, 1060 insertions(+)

diff --git a/distribution/pom.xml b/distribution/pom.xml
index e25b3c5..4ec9b26 100644
--- a/distribution/pom.xml
+++ b/distribution/pom.xml
@@ -324,6 +324,8 @@
 -c
 
org.apache.druid.extensions.contrib:druid-influx-extensions
 -c
+
org.apache.druid.extensions.contrib:druid-influxdb-emitter
+-c
 
org.apache.druid.extensions.contrib:druid-kafka-eight-simple-consumer
 -c
 
org.apache.druid.extensions.contrib:kafka-emitter
diff --git a/docs/content/development/extensions-contrib/influxdb-emitter.md 
b/docs/content/development/extensions-contrib/influxdb-emitter.md
new file mode 100644
index 000..138a0bb
--- /dev/null
+++ b/docs/content/development/extensions-contrib/influxdb-emitter.md
@@ -0,0 +1,75 @@
+---
+layout: doc_page
+title: "InfluxDB Emitter"
+---
+
+
+
+# InfluxDB Emitter
+
+To use this Apache Druid (incubating) extension, make sure to 
[include](../../operations/including-extensions.html) `druid-influxdb-emitter` 
extension.
+
+## Introduction
+
+This extension emits druid metrics to 
[InfluxDB](https://www.influxdata.com/time-series-platform/influxdb/) over 
HTTP. Currently this emitter only emits service metric events to InfluxDB (See 
[Druid metrics](../../operations/metrics.html) for a list of metrics).
+When a metric event is fired it is added to a queue of events. After a 
configurable amount of time, the events on the queue are transformed to 
InfluxDB's line protocol 
+and POSTed to the InfluxDB HTTP API. The entire queue is flushed at this 
point. The queue is also flushed as the emitter is shutdown.
+
+Note that authentication and authorization must be 
[enabled](https://docs.influxdata.com/influxdb/v1.7/administration/authentication_and_authorization/)
 on the InfluxDB server.
+
+## Configuration
+
+All the configuration parameters for the influxdb emitter are under 
`druid.emitter.influxdb`.
+
+|Property|Description|Required?|Default|
+||---|-|---|
+|`druid.emitter.influxdb.hostname`|The hostname of the InfluxDB 
server.|Yes|N/A|
+|`druid.emitter.influxdb.port`|The port of the InfluxDB server.|No|8086|
+|`druid.emitter.influxdb.databaseName`|The name of the database in 
InfluxDB.|Yes|N/A|
+|`druid.emitter.influxdb.maxQueueSize`|The size of the queue that holds 
events.|No|Integer.Max_Value(=2^31-1)|
+|`druid.emitter.influxdb.flushPeriod`|How often (in milliseconds) the events 
queue is parsed into Line Protocol and POSTed to InfluxDB.|No|6|
+|`druid.emitter.influxdb.flushDelay`|How long (in milliseconds) the scheduled 
method will wait until it first runs.|No|6|
+|`druid.emitter.influxdb.influxdbUserName`|The username for authenticating 
with the InfluxDB database.|Yes|N/A|
+|`druid.emitter.influxdb.influxdbPassword`|The password of the database 
authorized user|Yes|N/A|
+|`druid.emitter.influxdb.dimensionWhitelist`|A whitelist of metric dimensions 
to include as 
tags|No|`["dataSource","type","numMetrics","numDimensions","threshold","dimension","taskType","taskStatus","tier"]`|
+
+## InfluxDB Line Protocol
+
+An example of how this emitter parses a Druid metric event into InfluxDB's 
[lin

[incubator-druid] branch master updated (1fe0de1 -> bd899b9)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 1fe0de1  Fix currSize attribute of historical server type (#7706)
 add bd899b9  add s3 authentication method informations (#7674)

No new revisions were added by this update.

Summary of changes:
 docs/content/configuration/index.md|  5 ++--
 docs/content/development/extensions-core/s3.md | 39 --
 2 files changed, 34 insertions(+), 10 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: Virtual column updates for exploiting base column internal structure (#7618)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new fb0c846  Virtual column updates for exploiting base column internal 
structure (#7618)
fb0c846 is described below

commit fb0c84694143cda3e66d59e3d706f38f3fd5d1cb
Author: Himanshu 
AuthorDate: Mon May 20 17:04:35 2019 -0700

Virtual column updates for exploiting base column internal structure (#7618)

* VirtualColumn updates for exploiting base column internal structure

* unit tests for virtual column interface updates

* groupBy needs to use VirtualizedColumnSelectorFactory if outer query in
nested groupBy has virtual columns.

* fix strict compile checks

* fix teamcity build errors

* add comment explaining useVirtualizedColumnSelectorFactory flag in 
RowBasedGrouperHelper.createGrouperAccumulatorPair(..)
---
 .../epinephelinae/GroupByMergingQueryRunnerV2.java |   3 +-
 .../groupby/epinephelinae/GroupByRowProcessor.java |   6 +-
 .../epinephelinae/RowBasedGrouperHelper.java   |  29 +-
 .../query/groupby/strategy/GroupByStrategyV2.java  |   4 +-
 .../segment/ColumnSelectorBitmapIndexSelector.java |  55 ++-
 .../QueryableIndexColumnSelectorFactory.java   |  14 +-
 .../org/apache/druid/segment/VirtualColumn.java|  55 ++-
 .../org/apache/druid/segment/VirtualColumns.java   |  32 ++
 .../query/groupby/GroupByQueryRunnerTest.java  |   6 +-
 .../segment/virtual/DummyStringVirtualColumn.java  | 289 +++
 .../virtual/DummyStringVirtualColumnTest.java  | 395 +
 11 files changed, 865 insertions(+), 23 deletions(-)

diff --git 
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
 
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
index b8cc735..ef8c304 100644
--- 
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
+++ 
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunnerV2.java
@@ -201,7 +201,8 @@ public class GroupByMergingQueryRunnerV2 implements 
QueryRunner
   priority,
   hasTimeout,
   timeoutAt,
-  mergeBufferSize
+  mergeBufferSize,
+  false
   );
   final Grouper grouper = pair.lhs;
   final Accumulator accumulator = pair.rhs;
diff --git 
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
 
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
index 171b8ad..2fa3b57 100644
--- 
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
+++ 
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByRowProcessor.java
@@ -65,7 +65,8 @@ public class GroupByRowProcessor
   final String processingTmpDir,
   final int mergeBufferSize,
   final List closeOnExit,
-  final boolean wasQueryPushedDown
+  final boolean wasQueryPushedDown,
+  final boolean useVirtualizedColumnSelectorFactory
   )
   {
 final GroupByQuery query = (GroupByQuery) queryParam;
@@ -112,7 +113,8 @@ public class GroupByRowProcessor
 temporaryStorage,
 spillMapper,
 aggregatorFactories,
-mergeBufferSize
+mergeBufferSize,
+useVirtualizedColumnSelectorFactory
 );
 final Grouper grouper = pair.lhs;
 final Accumulator accumulator = pair.rhs;
diff --git 
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
 
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
index 0b82472..bd41d08 100644
--- 
a/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
+++ 
b/processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/RowBasedGrouperHelper.java
@@ -105,7 +105,8 @@ public class RowBasedGrouperHelper
   final LimitedTemporaryStorage temporaryStorage,
   final ObjectMapper spillMapper,
   final AggregatorFactory[] aggregatorFactories,
-  final int mergeBufferSize
+  final int mergeBufferSize,
+  final boolean useVirtualizedColumnSelectorFactory
   )
   {
 return createGrouperAccumulatorPair(
@@ -123,7 +124,8 @@ public class RowBasedGrouperHelper
 UNKNOWN_THREAD_PRIORITY,
 false,
 UNKNOWN_TIMEOUT,
-mergeBufferSize
+mergeBufferSize,
+useVirtualizedColumnSelectorFactory
 );
   }
 
@@ -147,7 +149,8 @@ public class RowBasedGrouperHelper
   final int priority,
   final boolean

[incubator-druid] branch master updated: make ComplexColumn an interface and ExtensionPoint (#7633)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 8687f42  make ComplexColumn an interface and ExtensionPoint (#7633)
8687f42 is described below

commit 8687f424f98a6570cfb732bf5f1b494366ff3112
Author: Himanshu 
AuthorDate: Wed May 15 20:59:55 2019 -0700

make ComplexColumn an interface and ExtensionPoint (#7633)

* make ComplexColumn an interface and ExtensionPoint

* incorporate review comments

* make ColumnValueSelector @ExtensionPoint

* more java docs

* add close() method to ComplexColumn interface
---
 .../segment/BaseDoubleColumnValueSelector.java |  4 +-
 .../segment/BaseFloatColumnValueSelector.java  |  4 +-
 .../druid/segment/BaseLongColumnValueSelector.java |  4 +-
 .../segment/BaseObjectColumnValueSelector.java |  4 +-
 .../apache/druid/segment/ColumnValueSelector.java  |  4 +-
 .../apache/druid/segment/column/ComplexColumn.java | 71 --
 java => GenericIndexedBasedComplexColumn.java} | 50 +--
 .../segment/serde/ComplexColumnPartSupplier.java   |  3 +-
 8 files changed, 66 insertions(+), 78 deletions(-)

diff --git 
a/processing/src/main/java/org/apache/druid/segment/BaseDoubleColumnValueSelector.java
 
b/processing/src/main/java/org/apache/druid/segment/BaseDoubleColumnValueSelector.java
index 42d38ae..8e1b9fc 100644
--- 
a/processing/src/main/java/org/apache/druid/segment/BaseDoubleColumnValueSelector.java
+++ 
b/processing/src/main/java/org/apache/druid/segment/BaseDoubleColumnValueSelector.java
@@ -19,7 +19,7 @@
 
 package org.apache.druid.segment;
 
-import org.apache.druid.guice.annotations.PublicApi;
+import org.apache.druid.guice.annotations.ExtensionPoint;
 import org.apache.druid.query.monomorphicprocessing.CalledFromHotLoop;
 import org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
 
@@ -30,7 +30,7 @@ import 
org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
  *
  * All implementations of this interface MUST also implement {@link 
ColumnValueSelector}.
  */
-@PublicApi
+@ExtensionPoint
 public interface BaseDoubleColumnValueSelector extends HotLoopCallee, 
BaseNullableColumnValueSelector
 {
   @CalledFromHotLoop
diff --git 
a/processing/src/main/java/org/apache/druid/segment/BaseFloatColumnValueSelector.java
 
b/processing/src/main/java/org/apache/druid/segment/BaseFloatColumnValueSelector.java
index bf70346..57c1138 100644
--- 
a/processing/src/main/java/org/apache/druid/segment/BaseFloatColumnValueSelector.java
+++ 
b/processing/src/main/java/org/apache/druid/segment/BaseFloatColumnValueSelector.java
@@ -19,7 +19,7 @@
 
 package org.apache.druid.segment;
 
-import org.apache.druid.guice.annotations.PublicApi;
+import org.apache.druid.guice.annotations.ExtensionPoint;
 import org.apache.druid.query.monomorphicprocessing.CalledFromHotLoop;
 import org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
 
@@ -30,7 +30,7 @@ import 
org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
  *
  * All implementations of this interface MUST also implement {@link 
ColumnValueSelector}.
  */
-@PublicApi
+@ExtensionPoint
 public interface BaseFloatColumnValueSelector extends HotLoopCallee, 
BaseNullableColumnValueSelector
 {
   @CalledFromHotLoop
diff --git 
a/processing/src/main/java/org/apache/druid/segment/BaseLongColumnValueSelector.java
 
b/processing/src/main/java/org/apache/druid/segment/BaseLongColumnValueSelector.java
index 1d95564..73525c7 100644
--- 
a/processing/src/main/java/org/apache/druid/segment/BaseLongColumnValueSelector.java
+++ 
b/processing/src/main/java/org/apache/druid/segment/BaseLongColumnValueSelector.java
@@ -19,7 +19,7 @@
 
 package org.apache.druid.segment;
 
-import org.apache.druid.guice.annotations.PublicApi;
+import org.apache.druid.guice.annotations.ExtensionPoint;
 import org.apache.druid.query.monomorphicprocessing.CalledFromHotLoop;
 import org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
 
@@ -30,7 +30,7 @@ import 
org.apache.druid.query.monomorphicprocessing.HotLoopCallee;
  *
  * All implementations of this interface MUST also implement {@link 
ColumnValueSelector}.
  */
-@PublicApi
+@ExtensionPoint
 public interface BaseLongColumnValueSelector extends HotLoopCallee, 
BaseNullableColumnValueSelector
 {
   @CalledFromHotLoop
diff --git 
a/processing/src/main/java/org/apache/druid/segment/BaseObjectColumnValueSelector.java
 
b/processing/src/main/java/org/apache/druid/segment/BaseObjectColumnValueSelector.java
index 45a399b..859f401 100644
--- 
a/processing/src/main/java/org/apache/druid/segment/BaseObjectColumnValueSelector.java
+++ 
b/processing/src/main/java/org/apache/druid/segment/BaseObjectColumnValueSelector.java
@@ -19,7 +19,7 @@
 
 package org.apache.druid.segment;
 
-imp

[incubator-druid] branch master updated: Remove Joda from indexing-hadoop (#7650)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new e8a6575  Remove Joda from indexing-hadoop (#7650)
e8a6575 is described below

commit e8a6575fb3dc1789aa450811aa6977497a5c85af
Author: Fokko Driesprong 
AuthorDate: Mon May 13 21:31:13 2019 +0200

Remove Joda from indexing-hadoop (#7650)
---
 indexing-hadoop/pom.xml | 4 
 1 file changed, 4 deletions(-)

diff --git a/indexing-hadoop/pom.xml b/indexing-hadoop/pom.xml
index 90f8f85..8716708 100644
--- a/indexing-hadoop/pom.xml
+++ b/indexing-hadoop/pom.xml
@@ -76,10 +76,6 @@
 jdbi
 
 
-joda-time
-joda-time
-
-
 com.google.code.findbugs
 jsr305
 


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated: add postgresql meta db table schema configuration property (#7137) (#7183)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 0ef435a  add postgresql meta db table schema configuration property 
(#7137) (#7183)
0ef435a is described below

commit 0ef435a16c511181bb97f61b230eeadf50d63535
Author: Jinseon Lee 
AuthorDate: Thu May 9 04:56:30 2019 +0900

add postgresql meta db table schema configuration property (#7137) (#7183)

* add postgresql meta db table schema configuration property (#7137)

If the postgresql db schema changes, you must set the configuration
values.
You do not need to set it if there is no change from the default schema
'public'.
druid.metadata.postgres.dbTableSchema=public

* create postgresql metadb table schema configuration property (#7137)
If the postgresql db schema changes, you must set the configuration
values.
You do not need to set it if there is no change from the default schema
'public'.
druid.metadata.postgres.dbTableSchema=public
check PostgreSQLTablesConfig.java

* modify postgresql readme file. - metadb table schema (#7137)
If the postgresql db schema changes, you must set the configuration
values.
You do not need to set it if there is no change from the default schema
'public'.
druid.metadata.postgres.dbTableSchema=public
check PostgreSQLTablesConfig.java
---
 .../development/extensions-core/postgresql.md  |  2 ++
 .../storage/postgresql/PostgreSQLConnector.java| 11 --
 .../PostgreSQLMetadataStorageModule.java   |  1 +
 .../storage/postgresql/PostgreSQLTablesConfig.java | 41 ++
 .../postgresql/PostgreSQLConnectorTest.java|  3 +-
 5 files changed, 54 insertions(+), 4 deletions(-)

diff --git a/docs/content/development/extensions-core/postgresql.md 
b/docs/content/development/extensions-core/postgresql.md
index 07a2a78..26f77fc 100644
--- a/docs/content/development/extensions-core/postgresql.md
+++ b/docs/content/development/extensions-core/postgresql.md
@@ -83,3 +83,5 @@ In most cases, the configuration options map directly to the 
[postgres jdbc conn
 | `druid.metadata.postgres.ssl.sslRootCert` | The full path to the root 
certificate. | none | no |
 | `druid.metadata.postgres.ssl.sslHostNameVerifier` | The classname of the 
hostname verifier. | none | no |
 | `druid.metadata.postgres.ssl.sslPasswordCallback` | The classname of the SSL 
password provider. | none | no |
+| `druid.metadata.postgres.dbTableSchema` | druid meta table schema | `public` 
| no |
+
diff --git 
a/extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/metadata/storage/postgresql/PostgreSQLConnector.java
 
b/extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/metadata/storage/postgresql/PostgreSQLConnector.java
index e234a15..a474a0b 100644
--- 
a/extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/metadata/storage/postgresql/PostgreSQLConnector.java
+++ 
b/extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/metadata/storage/postgresql/PostgreSQLConnector.java
@@ -48,11 +48,14 @@ public class PostgreSQLConnector extends 
SQLMetadataConnector
 
   private volatile Boolean canUpsert;
 
+  private final String dbTableSchema;
+  
   @Inject
   public PostgreSQLConnector(
   Supplier config,
   Supplier dbTables,
-  PostgreSQLConnectorConfig connectorConfig
+  PostgreSQLConnectorConfig connectorConfig,
+  PostgreSQLTablesConfig tablesConfig
   )
   {
 super(config, dbTables);
@@ -104,7 +107,8 @@ public class PostgreSQLConnector extends 
SQLMetadataConnector
 }
 
 this.dbi = new DBI(datasource);
-
+this.dbTableSchema = tablesConfig.getDbTableSchema();
+
 log.info("Configured PostgreSQL as metadata storage");
   }
 
@@ -146,8 +150,9 @@ public class PostgreSQLConnector extends 
SQLMetadataConnector
   public boolean tableExists(final Handle handle, final String tableName)
   {
 return !handle.createQuery(
-"SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 
'public' AND tablename ILIKE :tableName"
+"SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 
:dbTableSchema AND tablename ILIKE :tableName"
 )
+  .bind("dbTableSchema", dbTableSchema)
   .bind("tableName", tableName)
   .map(StringMapper.FIRST)
   .list()
diff --git 
a/extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/metadata/storage/postgresql/PostgreSQLMetadataStorageModule.java
 
b/extensions-core/postgresql-metadata-storage/src/main/java/org/apache/druid/metadata/storage/postgresql/PostgreSQLMetadataStorageModule.java
index f10de65..9506edd 100644
--- 

[incubator-druid] branch master updated: updated to use the latest sketches-core-0.12.0 (#6381)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new ec9d182  updated to use the latest sketches-core-0.12.0 (#6381)
ec9d182 is described below

commit ec9d1827a05f4179f2189eac32287e14287d6126
Author: Alexander Saydakov 
AuthorDate: Tue Oct 23 11:20:19 2018 -0700

updated to use the latest sketches-core-0.12.0 (#6381)
---
 extensions-core/datasketches/pom.xml   |   2 +-
 .../hll/HllSketchBuildBufferAggregator.java|   3 +-
 .../hll/HllSketchMergeBufferAggregator.java|   7 +-
 .../datasketches/hll/HllSketchObjectStrategy.java  |   3 +-
 .../DoublesSketchBuildBufferAggregator.java|   3 +-
 .../DoublesSketchMergeBufferAggregator.java|   3 +-
 .../quantiles/DoublesSketchObjectStrategy.java |   4 +-
 .../datasketches/theta/SketchAggregator.java   |  14 ++-
 .../datasketches/theta/SketchBufferAggregator.java |   3 +-
 .../theta/SketchHolderObjectStrategy.java  |   3 +-
 .../datasketches/theta/SynchronizedUnion.java  | 133 -
 .../ArrayOfDoublesSketchBuildBufferAggregator.java |   7 +-
 .../ArrayOfDoublesSketchMergeBufferAggregator.java |   7 +-
 .../tuple/ArrayOfDoublesSketchObjectStrategy.java  |   3 +-
 .../tuple/ArrayOfDoublesSketchAggregationTest.java |   1 +
 15 files changed, 39 insertions(+), 157 deletions(-)

diff --git a/extensions-core/datasketches/pom.xml 
b/extensions-core/datasketches/pom.xml
index 1abb026..ef44977 100644
--- a/extensions-core/datasketches/pom.xml
+++ b/extensions-core/datasketches/pom.xml
@@ -38,7 +38,7 @@
 
   com.yahoo.datasketches
   sketches-core
-  0.10.3
+  0.12.0
 
 
   org.apache.commons
diff --git 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregator.java
 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregator.java
index cab8019..bd2d047 100644
--- 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregator.java
+++ 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregator.java
@@ -20,6 +20,7 @@
 package org.apache.druid.query.aggregation.datasketches.hll;
 
 import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
 import java.util.IdentityHashMap;
 import java.util.concurrent.locks.Lock;
 import java.util.concurrent.locks.ReadWriteLock;
@@ -135,7 +136,7 @@ public class HllSketchBuildBufferAggregator implements 
BufferAggregator
 
   private WritableMemory getMemory(final ByteBuffer buf)
   {
-return memCache.computeIfAbsent(buf, b -> WritableMemory.wrap(b));
+return memCache.computeIfAbsent(buf, b -> WritableMemory.wrap(b, 
ByteOrder.LITTLE_ENDIAN));
   }
 
   /**
diff --git 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchMergeBufferAggregator.java
 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchMergeBufferAggregator.java
index 3477bb0..8e72d23 100644
--- 
a/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchMergeBufferAggregator.java
+++ 
b/extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchMergeBufferAggregator.java
@@ -20,6 +20,7 @@
 package org.apache.druid.query.aggregation.datasketches.hll;
 
 import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
 import java.util.concurrent.locks.Lock;
 import java.util.concurrent.locks.ReadWriteLock;
 
@@ -66,7 +67,7 @@ public class HllSketchMergeBufferAggregator implements 
BufferAggregator
   @Override
   public void init(final ByteBuffer buf, final int position)
   {
-final WritableMemory mem = 
WritableMemory.wrap(buf).writableRegion(position, size);
+final WritableMemory mem = WritableMemory.wrap(buf, 
ByteOrder.LITTLE_ENDIAN).writableRegion(position, size);
 // Not necessary to keep the constructed object since it is cheap to 
reconstruct by wrapping the memory.
 // The objects are not cached as in BuildBufferAggregator since they never 
exceed the max size and never move.
 // So it is easier to reconstruct them by wrapping memory then to keep 
position-to-object mappings. 
@@ -85,7 +86,7 @@ public class HllSketchMergeBufferAggregator implements 
BufferAggregator
 if (sketch == null) {
   return;
 }
-final WritableMemory mem = 
WritableMemory.wrap(buf).writableRegion(position, size);
+final WritableMemory mem = WritableMemory.wrap(buf, 
ByteOrder.LITTLE_ENDIAN).writableRegion(position, size);
 final Lock lock = 
stripedLock

[incubator-druid] branch master updated: Securing passwords used for SSL connections to Kafka (#6285)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new ab7b479  Securing passwords used for SSL connections to Kafka (#6285)
ab7b479 is described below

commit ab7b4798cc2747bb68550b67f69660965a276dc8
Author: Atul Mohan 
AuthorDate: Thu Oct 11 12:03:01 2018 -0500

Securing passwords used for SSL connections to Kafka (#6285)

* Secure credentials in consumer properties

* Merge master

* Refactor property population into separate method

* Fix property setter

* Fix tests
---
 .../development/extensions-core/kafka-ingestion.md |  2 +-
 .../apache/druid/indexing/kafka/KafkaIOConfig.java |  6 ++--
 .../druid/indexing/kafka/KafkaIndexTask.java   | 32 ++
 .../indexing/kafka/supervisor/KafkaSupervisor.java |  7 +++--
 .../kafka/supervisor/KafkaSupervisorIOConfig.java  |  9 --
 .../druid/indexing/kafka/KafkaIndexTaskTest.java   | 12 
 .../supervisor/KafkaSupervisorIOConfigTest.java| 28 +++
 .../kafka/supervisor/KafkaSupervisorTest.java  |  5 ++--
 .../druid/indexing/kafka/test/TestBroker.java  |  4 +--
 9 files changed, 81 insertions(+), 24 deletions(-)

diff --git a/docs/content/development/extensions-core/kafka-ingestion.md 
b/docs/content/development/extensions-core/kafka-ingestion.md
index 568fc94..12bd5f6 100644
--- a/docs/content/development/extensions-core/kafka-ingestion.md
+++ b/docs/content/development/extensions-core/kafka-ingestion.md
@@ -166,7 +166,7 @@ For Roaring bitmaps:
 |Field|Type|Description|Required|
 |-||---||
 |`topic`|String|The Kafka topic to read from. This must be a specific topic as 
topic patterns are not supported.|yes|
-|`consumerProperties`|Map|A map of properties to be passed to 
the Kafka consumer. This must contain a property `bootstrap.servers` with a 
list of Kafka brokers in the form: 
`:,:,...`.|yes|
+|`consumerProperties`|Map|A map of properties to be passed to 
the Kafka consumer. This must contain a property `bootstrap.servers` with a 
list of Kafka brokers in the form: 
`:,:,...`. For SSL connections, the 
`keystore`, `truststore` and `key` passwords can be provided as a [Password 
Provider](../../operations/password-provider.html) or String password.|yes|
 |`replicas`|Integer|The number of replica sets, where 1 means a single set of 
tasks (no replication). Replica tasks will always be assigned to different 
workers to provide resiliency against node failure.|no (default == 1)|
 |`taskCount`|Integer|The maximum number of *reading* tasks in a *replica set*. 
This means that the maximum number of reading tasks will be `taskCount * 
replicas` and the total number of tasks (*reading* + *publishing*) will be 
higher than this. See 'Capacity Planning' below for more details. The number of 
reading tasks will be less than `taskCount` if `taskCount > 
{numKafkaPartitions}`.|no (default == 1)|
 |`taskDuration`|ISO8601 Period|The length of time before tasks stop reading 
and begin publishing their segment.|no (default == PT1H)|
diff --git 
a/extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaIOConfig.java
 
b/extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaIOConfig.java
index 3c60449..6a9af7f 100644
--- 
a/extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaIOConfig.java
+++ 
b/extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaIOConfig.java
@@ -39,7 +39,7 @@ public class KafkaIOConfig implements IOConfig
   private final String baseSequenceName;
   private final KafkaPartitions startPartitions;
   private final KafkaPartitions endPartitions;
-  private final Map consumerProperties;
+  private final Map consumerProperties;
   private final boolean useTransaction;
   private final Optional minimumMessageTime;
   private final Optional maximumMessageTime;
@@ -51,7 +51,7 @@ public class KafkaIOConfig implements IOConfig
   @JsonProperty("baseSequenceName") String baseSequenceName,
   @JsonProperty("startPartitions") KafkaPartitions startPartitions,
   @JsonProperty("endPartitions") KafkaPartitions endPartitions,
-  @JsonProperty("consumerProperties") Map 
consumerProperties,
+  @JsonProperty("consumerProperties") Map 
consumerProperties,
   @JsonProperty("useTransaction") Boolean useTransaction,
   @JsonProperty("minimumMessageTime") DateTime minimumMessageTime,
   @JsonProperty("maximumMessageTime") DateTime maximumMessageTime,
@@ -114,7 +114,7 @@ public class KafkaIOConfig implements IOConfig
   }
 
   @JsonProperty
-  public Map getConsumerProperties()
+  public Map getCons

[incubator-druid] branch master updated: Add ability to specify list of task ports and port range (#6263)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new 87ccee0  Add ability to specify list of task ports and port range 
(#6263)
87ccee0 is described below

commit 87ccee05f76e983874b41090f2b0a6fe1cf4b8dd
Author: QiuMM 
AuthorDate: Fri Sep 14 10:36:04 2018 +0800

Add ability to specify list of task ports and port range (#6263)

* support specify list of task ports

* fix typos

* address comments

* remove druid.indexer.runner.separateIngestionEndpoint config

* tweak doc

* fix doc

* code cleanup

* keep some useful comments
---
 docs/content/configuration/index.md| 13 ++---
 .../druid/indexing/overlord/ForkingTaskRunner.java | 30 +--
 .../apache/druid/indexing/overlord/PortFinder.java | 54 +++
 .../overlord/config/ForkingTaskRunnerConfig.java   | 31 ---
 .../druid/indexing/overlord/PortFinderTest.java| 60 ++
 .../config/ForkingTaskRunnerConfigTest.java| 25 +
 .../jetty/ChatHandlerServerModule.java | 33 ++--
 7 files changed, 129 insertions(+), 117 deletions(-)

diff --git a/docs/content/configuration/index.md 
b/docs/content/configuration/index.md
index f430db7..771b872 100644
--- a/docs/content/configuration/index.md
+++ b/docs/content/configuration/index.md
@@ -1039,8 +1039,9 @@ Middle managers pass their configurations down to their 
child peons. The middle
 |`druid.indexer.runner.javaOpts`|*DEPRECATED* A string of -X Java options to 
pass to the peon's JVM. Quotable parameters or parameters with spaces are 
encouraged to use javaOptsArray|""|
 |`druid.indexer.runner.javaOptsArray`|A json array of strings to be passed in 
as options to the peon's jvm. This is additive to javaOpts and is recommended 
for properly handling arguments which contain quotes or spaces like 
`["-XX:OnOutOfMemoryError=kill -9 %p"]`|`[]`|
 |`druid.indexer.runner.maxZnodeBytes`|The maximum size Znode in bytes that can 
be created in Zookeeper.|524288|
-|`druid.indexer.runner.startPort`|Starting port used for peon processes, 
should be greater than 1023.|8100|
-|`druid.indexer.runner.separateIngestionEndpoint`|*Deprecated.* Use separate 
server and consequently separate jetty thread pool for ingesting events. Not 
supported with TLS.|false|
+|`druid.indexer.runner.startPort`|Starting port used for peon processes, 
should be greater than 1023 and less than 65536.|8100|
+|`druid.indexer.runner.endPort`|Ending port used for peon processes, should be 
greater than or equal to `druid.indexer.runner.startPort` and less than 
65536.|65535|
+|`druid.indexer.runner.ports`|A json array of integers to specify ports that 
used for peon processes. If provided and non-empty, ports for peon processes 
will be chosen from these ports. And 
`druid.indexer.runner.startPort/druid.indexer.runner.endPort` will be 
completely ignored.|`[]`|
 |`druid.worker.ip`|The IP of the worker.|localhost|
 |`druid.worker.version`|Version identifier for the middle manager.|0|
 |`druid.worker.capacity`|Maximum number of tasks the middle manager can 
accept.|Number of available processors - 1|
@@ -1103,14 +1104,6 @@ Additional peon configs include:
 |`druid.indexer.task.restoreTasksOnRestart`|If true, middleManagers will 
attempt to stop tasks gracefully on shutdown and restore them on restart.|false|
 |`druid.indexer.server.maxChatRequests`|Maximum number of concurrent requests 
served by a task's chat handler. Set to 0 to disable limiting.|0|
 
-If the deprecated `druid.indexer.runner.separateIngestionEndpoint` property is 
set to true then following configurations
-are available for the ingestion server at peon:
-
-|Property|Description|Default|
-||---|---|
-|`druid.indexer.server.chathandler.http.numThreads`|*Deprecated.* Number of 
threads for HTTP requests.|Math.max(10, (Number of available processors * 17) / 
16 + 2) + 30|
-|`druid.indexer.server.chathandler.http.maxIdleTime`|*Deprecated.* The Jetty 
max idle time for a connection.|PT5M|
-
 If the peon is running in remote mode, there must be an overlord up and 
running. Peons in remote mode can set the following configurations:
 
 |Property|Description|Default|
diff --git 
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java
 
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java
index 05bf2a1..c393c41 100644
--- 
a/indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java
+++ 
b/indexing-service/src/main/java/org/apache/druid/indexing/overlord/ForkingTaskRunner.java
@@ -128,7 +128,7 @@ public class ForkingTaskRunner implements TaskRunner, 
TaskLogStreamer
 this.taskLogPu

[incubator-druid] branch master updated: remove unnecessary tlsPortFinder to avoid potential port conflicts (#6194)

[himanshug]

This is an automated email from the ASF dual-hosted git repository.

himanshug pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git


The following commit(s) were added to refs/heads/master by this push:
 new ceb8f8e  remove unnecessary tlsPortFinder to avoid potential port 
conflicts (#6194)
ceb8f8e is described below

commit ceb8f8e6258e1d113d8162a9a8a2fc62810704eb
Author: QiuMM 
AuthorDate: Fri Aug 24 01:41:49 2018 +0800

remove unnecessary tlsPortFinder to avoid potential port conflicts (#6194)
---
 docs/content/configuration/index.md|  1 -
 .../java/io/druid/indexing/overlord/ForkingTaskRunner.java |  6 ++
 .../indexing/overlord/config/ForkingTaskRunnerConfig.java  | 10 --
 3 files changed, 2 insertions(+), 15 deletions(-)

diff --git a/docs/content/configuration/index.md 
b/docs/content/configuration/index.md
index 5eb03a6..5f970c0 100644
--- a/docs/content/configuration/index.md
+++ b/docs/content/configuration/index.md
@@ -1040,7 +1040,6 @@ Middle managers pass their configurations down to their 
child peons. The middle
 |`druid.indexer.runner.javaOptsArray`|A json array of strings to be passed in 
as options to the peon's jvm. This is additive to javaOpts and is recommended 
for properly handling arguments which contain quotes or spaces like 
`["-XX:OnOutOfMemoryError=kill -9 %p"]`|`[]`|
 |`druid.indexer.runner.maxZnodeBytes`|The maximum size Znode in bytes that can 
be created in Zookeeper.|524288|
 |`druid.indexer.runner.startPort`|Starting port used for peon processes, 
should be greater than 1023.|8100|
-|`druid.indexer.runner.tlsStartPort`|Starting TLS port for peon processes, 
should be greater than 1023.|8300|
 |`druid.indexer.runner.separateIngestionEndpoint`|*Deprecated.* Use separate 
server and consequently separate jetty thread pool for ingesting events. Not 
supported with TLS.|false|
 |`druid.worker.ip`|The IP of the worker.|localhost|
 |`druid.worker.version`|Version identifier for the middle manager.|0|
diff --git 
a/indexing-service/src/main/java/io/druid/indexing/overlord/ForkingTaskRunner.java
 
b/indexing-service/src/main/java/io/druid/indexing/overlord/ForkingTaskRunner.java
index 6326fc0..37462d2 100644
--- 
a/indexing-service/src/main/java/io/druid/indexing/overlord/ForkingTaskRunner.java
+++ 
b/indexing-service/src/main/java/io/druid/indexing/overlord/ForkingTaskRunner.java
@@ -104,7 +104,6 @@ public class ForkingTaskRunner implements TaskRunner, 
TaskLogStreamer
   private final ListeningExecutorService exec;
   private final ObjectMapper jsonMapper;
   private final PortFinder portFinder;
-  private final PortFinder tlsPortFinder;
   private final CopyOnWriteArrayList> 
listeners = new CopyOnWriteArrayList<>();
 
   // Writes must be synchronized. This is only a ConcurrentMap so 
"informational" reads can occur without waiting.
@@ -130,7 +129,6 @@ public class ForkingTaskRunner implements TaskRunner, 
TaskLogStreamer
 this.jsonMapper = jsonMapper;
 this.node = node;
 this.portFinder = new PortFinder(config.getStartPort());
-this.tlsPortFinder = new PortFinder(config.getTlsStartPort());
 this.exec = MoreExecutors.listeningDecorator(
 Execs.multiThreaded(workerConfig.getCapacity(), 
"forking-task-runner-%d")
 );
@@ -247,7 +245,7 @@ public class ForkingTaskRunner implements TaskRunner, 
TaskLogStreamer
 }
 
 if (node.isEnableTlsPort()) {
-  tlsChildPort = tlsPortFinder.findUnusedPort();
+  tlsChildPort = portFinder.findUnusedPort();
 }
 
 final TaskLocation taskLocation = 
TaskLocation.create(childHost, childPort, tlsChildPort);
@@ -515,7 +513,7 @@ public class ForkingTaskRunner implements TaskRunner, 
TaskLogStreamer
   portFinder.markPortUnused(childPort);
 }
 if (node.isEnableTlsPort()) {
-  tlsPortFinder.markPortUnused(tlsChildPort);
+  portFinder.markPortUnused(tlsChildPort);
 }
 if (childChatHandlerPort > 0) {
   portFinder.markPortUnused(childChatHandlerPort);
diff --git 
a/indexing-service/src/main/java/io/druid/indexing/overlord/config/ForkingTaskRunnerConfig.java
 
b/indexing-service/src/main/java/io/druid/indexing/overlord/config/ForkingTaskRunnerConfig.java
index fbe171e..4c1bfd1 100644
--- 
a/indexing-service/src/main/java/io/druid/indexing/overlord/config/ForkingTaskRunnerConfig.java
+++ 
b/indexing-service/src/main/java/io/druid/indexing/overlord/config/ForkingTaskRunnerConfig.java
@@ -63,11 +63,6 @@ public class ForkingTaskRunnerConfig
   private int startPort = 8100;
 
   @JsonProperty
-  @Min(1024)
-  @Max