[GitHub] [incubator-druid] stale[bot] closed issue #6447: Bug: Zero filling time buckets below the queryGranularity
stale[bot] closed issue #6447: Bug: Zero filling time buckets below the queryGranularity URL: https://github.com/apache/incubator-druid/issues/6447 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed issue #6446: Insertion failing in druid_config but it is logged in druid_audit?
stale[bot] closed issue #6446: Insertion failing in druid_config but it is logged in druid_audit? URL: https://github.com/apache/incubator-druid/issues/6446 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6447: Bug: Zero filling time buckets below the queryGranularity
stale[bot] commented on issue #6447: Bug: Zero filling time buckets below the queryGranularity URL: https://github.com/apache/incubator-druid/issues/6447#issuecomment-521634327 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6446: Insertion failing in druid_config but it is logged in druid_audit?
stale[bot] commented on issue #6446: Insertion failing in druid_config but it is logged in druid_audit? URL: https://github.com/apache/incubator-druid/issues/6446#issuecomment-521634322 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #8310: exclude kerberos extension dependencies that are already included in druid core libraries
clintropolis commented on issue #8310: exclude kerberos extension dependencies that are already included in druid core libraries URL: https://github.com/apache/incubator-druid/pull/8310#issuecomment-521563375 Decided to just go ahead and try removing most of the ones intellij indicated we shouldn't need: https://user-images.githubusercontent.com/1577461/63083424-8b9f7f80-befd-11e9-8a2d-f73686bd185d.png;> This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pphust commented on issue #8316: fix issue8291: make sure ReferenceCountingSegment.decrement() is invoked correctly when useCache=true
pphust commented on issue #8316: fix issue8291: make sure ReferenceCountingSegment.decrement() is invoked correctly when useCache=true URL: https://github.com/apache/incubator-druid/pull/8316#issuecomment-521625202 The PR is submitted for 0.14.2. But I guess the problem exist in 0.15.x also This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pphust commented on issue #8291: ReferenceCountedSegment is acquired too many times
pphust commented on issue #8291: ReferenceCountedSegment is acquired too many times URL: https://github.com/apache/incubator-druid/issues/8291#issuecomment-521624905 I submit https://github.com/apache/incubator-druid/pull/8316 to fix the problem This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] suntyu commented on issue #8312: Druid Jetty Exception
suntyu commented on issue #8312: Druid Jetty Exception URL: https://github.com/apache/incubator-druid/issues/8312#issuecomment-521631635 1.hadoop2.6.0-cdh5.15; 2.all web-ui is 500; This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] pphust opened a new pull request #8316: fix issue8291: make sure ReferenceCountingSegment.decrement() is invoked correctly when useCache=true
pphust opened a new pull request #8316: fix issue8291: make sure ReferenceCountingSegment.decrement() is invoked correctly when useCache=true URL: https://github.com/apache/incubator-druid/pull/8316 Issue description: The issue8291 is submitted by my colleague. It will cause ReferenceCountingSegment only invoke _**increase()**_ but do not invoke _**decrease()**_ correspondly. Then _**getNumReferences()**_ will keep increasing and meet 65535 limit. Finally it will throw exception and cause all query fail. Root reason: The problem only happens when useCache=true in both middle manager node setting and query context. And cache type and cache strategy should meet some requirement also. In this scenario, actually SinkQuerySegmentWalker will use CachingQueryRunner for querying. As designed, it will invoke decrease() after the background queryRunner.run() is invoked. All looks perfect. But when useCache=true, CachingQueryRunner just retrieve query result from cache directly and queryRunner.run() is NOT invoked really. Then of course decrease() will not be invoked also. It is a problem. Solution: make sure decrease() will be invoked in any case. How we make it in pull request: 1、add one protected EMPTY close() in useCache=true statement of CachingQueryRunner， then it will not impact other features which still use CachingQueryRunner. 2、add CloseableCachingQueryRunner to extend CachingQueryRunner. It will accept one extra Closeable parameter in constructor. 3、overwrite close() in CloseableCachingQueryRunner, in which it will use Closeable to invoke decrease(). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] somanullah commented on issue #7741: Reindexing tasks containing thetaSketches resulting in incorrect sketch values
somanullah commented on issue #7741: Reindexing tasks containing thetaSketches resulting in incorrect sketch values URL: https://github.com/apache/incubator-druid/issues/7741#issuecomment-521641969 @a2l007 @leventov We're seeing similar behavior on our cluster. It was recently upgraded to 0.14.0 and we have noticed differences in sketch results for the same data when comparing with data indexed from 0.12.1. Is there a fix for this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ThirdLibResearch opened a new issue #8315: Third-Party Library Version Inconsistencies in your project
ThirdLibResearch opened a new issue #8315: Third-Party Library Version Inconsistencies in your project URL: https://github.com/apache/incubator-druid/issues/8315 We are currently investigating the usage of different versions of the same third-party library in different modules of a project (i.e., the library version inconsistency problem). It is a non-trivial and time-consuming task to maintain the evolving third-party libraries. In that sense, it seems necessary to harmonize the different versions of the same third-party library in different modules into one single version. Thus we developed a prototype tool to detect library version inconsistencies and suggest harmonized version with detailed maintenance efforts. We applied our tool on your GitHub project, and attached the generated report. We hope that the report could be useful for you to be aware of the library version inconsistencies in your project and to decide whether to harmonize inconsistent library versions. Here is the report: [druid-io druid.pdf](https://github.com/ThirdLibResearch/ThirdLibSource/blob/master/issues/report/druid-io%20druid.pdf) Some pictures may be too large to present in the pdf so we attach them below: [692_junitjunitjar_0.png](https://raw.githubusercontent.com/ThirdLibResearch/ThirdLibSource/master/issues/image/692_junitjunitjar_0.png) [692_org.easymockeasymockjar_0.png](https://raw.githubusercontent.com/ThirdLibResearch/ThirdLibSource/master/issues/image/692_org.easymockeasymockjar_0.png) We will appreciate your comments very much. Best regards This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] asdf2014 opened a new pull request #8313: Hide descriptive comments in `pull_request_template.md`
asdf2014 opened a new pull request #8313: Hide descriptive comments in `pull_request_template.md` URL: https://github.com/apache/incubator-druid/pull/8313 ### Description Hide descriptive comments in `pull_request_template.md`. This PR has: - [x] been self-reviewed. - [ ] using the [concurrency checklist](https://github.com/apache/incubator-druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.) - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6016: Druid 'Shapeshifting' Columns
stale[bot] commented on issue #6016: Druid 'Shapeshifting' Columns URL: https://github.com/apache/incubator-druid/pull/6016#issuecomment-521526674 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #6345: Druid-Kafka-Indexing-Service Running Time
stale[bot] commented on issue #6345: Druid-Kafka-Indexing-Service Running Time URL: https://github.com/apache/incubator-druid/issues/6345#issuecomment-521526694 This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] closed issue #6345: Druid-Kafka-Indexing-Service Running Time
stale[bot] closed issue #6345: Druid-Kafka-Indexing-Service Running Time URL: https://github.com/apache/incubator-druid/issues/6345 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Web-Console: add more side column functions to query view (#8283)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 8924d28 Web-Console: add more side column functions to query view
(#8283)
8924d28 is described below
commit 8924d285dc39dc548db611aa4429ae5e62d64a44
Author: mcbrewster <37322608+mcbrews...@users.noreply.github.com>
AuthorDate: Thu Aug 15 01:22:25 2019 -0700
Web-Console: add more side column functions to query view (#8283)
* small fixes
* run jest -u
* add trim
* add to groupby
* save
* add functionality to column tree menu
* remove history dialog
* rename file
* fixes
* add new values to time-menu
* add query dialog
* add more functions to collumn-tree menu
* add menu functions
* remove test
* update tests, fix extra 0s, clear in between time filters
* fixes
* add deffered
* save
* add alias's
* change menu style
* fix substring
* small fixes
* add auto run to run button
* move divider to a reasonable position
* fix auto expland
* run jest -u
* fix auto run
* prevent aggregate cooumn changes
* fix group by bug
* add quertAst to state
* ui fixes
* prevent filter on non existant ast
* add auto run to shows
* use handle run
* handle *
* add alias and tslint fix
---
web-console/package-lock.json | 6 +-
web-console/package.json | 2 +-
.../deferred/__snapshots__/deferred.spec.tsx.snap | 3 +
.../deferred/deferred.spec.tsx}| 13 +-
.../deferred/deferred.tsx} | 26 +-
.../query-plan-history.spec.tsx.snap} | 35 +-
.../query-history-dialog.scss} | 25 +-
.../query-history-dialog/query-history-dialog.tsx | 91 +
.../query-plan-history.spec.tsx} | 4 +-
.../__snapshots__/query-plan-dialog.spec.tsx.snap | 10 +
.../query-plan-dialog/query-plan-dialog.spec.tsx | 7 +-
.../query-plan-dialog/query-plan-dialog.tsx| 34 +-
web-console/src/utils/local-storage-keys.tsx | 2 +
.../__snapshots__/query-view.spec.tsx.snap | 12 +-
.../__snapshots__/number-menu-items.spec.tsx.snap | 3 +
.../number-menu-items/number-menu-items.spec.tsx} | 21 +-
.../number-menu-items/number-menu-items.tsx| 141
.../__snapshots__/string-menu-items.spec.tsx.snap | 3 +
.../string-menu-items/string-menu-items.spec.tsx} | 21 +-
.../string-menu-items/string-menu-items.tsx| 161 +
.../__snapshots__/time-menu-items.spec.tsx.snap| 3 +
.../time-menu-items/time-menu-items.spec.tsx} | 22 +-
.../time-menu-items/time-menu-items.tsx| 380 +
.../query-view/column-tree/column-tree.spec.tsx| 7 +
.../views/query-view/column-tree/column-tree.tsx | 155 ++---
.../query-view/query-output/query-output.spec.tsx | 1 -
.../views/query-view/query-output/query-output.tsx | 164 -
web-console/src/views/query-view/query-view.tsx| 347 +++
.../query-view/run-button/run-button.spec.tsx | 3 +
.../src/views/query-view/run-button/run-button.tsx | 20 +-
30 files changed, 1451 insertions(+), 271 deletions(-)
diff --git a/web-console/package-lock.json b/web-console/package-lock.json
index fdac733..5c25e00 100644
--- a/web-console/package-lock.json
+++ b/web-console/package-lock.json
@@ -4395,9 +4395,9 @@
"integrity":
"sha512-0sYnfUHHMoajaud/i5BHKA12bUxiWEHJ9rxGqVEppFxsEcxef0TZQ5J59lU+UniEBcz/sG5fTESRyS7cOm3tSQ=="
},
"druid-query-toolkit": {
- "version": "0.3.15",
- "resolved":
"https://registry.npmjs.org/druid-query-toolkit/-/druid-query-toolkit-0.3.15.tgz;,
- "integrity":
"sha512-q7uKfUdBItjOyNF1PlWF/rAhOim1uAjI085fsoKIBDZ2o5O4XRjaCKqXtW49Ovv92ks/22zLoYWNdU51i4PB/w==",
+ "version": "0.3.20",
+ "resolved":
"https://registry.npmjs.org/druid-query-toolkit/-/druid-query-toolkit-0.3.20.tgz;,
+ "integrity":
"sha512-jrGNu+o/nD+uhbxAMLXEQrSWNEylCRmkiuFDJSPCMz7cjMNArsdIgyBQHPezNAeTDpReelAt59xJ7pvqXwPIvw==",
"requires": {
"tslib": "^1.10.0"
}
diff --git a/web-console/package.json b/web-console/package.json
index 14ef194..8f3b214 100644
--- a/web-console/package.json
+++ b/web-console/package.json
@@ -61,7 +61,7 @@
"d3": "^5.9.7",
"d3-array": "^2.2.0",
"druid-console": "^0.0.2",
-"druid-query-toolkit": "^0.3.15",
+"druid-query-toolkit": "^0.3.20",
"file-saver": "^2.0.2",
"has-own-prop": "^2.0.0",
"hjson": "^3.1.2",
diff --git
[GitHub] [incubator-druid] clintropolis merged pull request #8283: Web-Console: add more side column functions to query view
clintropolis merged pull request #8283: Web-Console: add more side column functions to query view URL: https://github.com/apache/incubator-druid/pull/8283 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] suntyu opened a new issue #8312: druid jetty Eception
suntyu opened a new issue #8312: druid jetty Eception URL: https://github.com/apache/incubator-druid/issues/8312 #java.util.concurrent.TimeoutException: Idle timeout expired: 30/30 ms #Need to specify class name in environment or system property, or as an applet parameter, or in an application resource file: java.naming.factory.ini tial #org.eclipse.jetty.http.BadMessageException: 500: No version This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8257: Add support for parallel native indexing with shuffle for perfect rollup
clintropolis commented on a change in pull request #8257: Add support for
parallel native indexing with shuffle for perfect rollup
URL: https://github.com/apache/incubator-druid/pull/8257#discussion_r314207977
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialSegmentMergeTask.java
##
@@ -0,0 +1,453 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.common.task.batch.parallel;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.base.Stopwatch;
+import it.unimi.dsi.fastutil.ints.Int2ObjectMap;
+import it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap;
+import org.apache.commons.io.FileUtils;
+import org.apache.druid.client.indexing.IndexingServiceClient;
+import org.apache.druid.data.input.impl.prefetch.Fetchers;
+import org.apache.druid.indexer.TaskStatus;
+import org.apache.druid.indexer.partitions.HashedPartitionsSpec;
+import org.apache.druid.indexing.common.TaskLock;
+import org.apache.druid.indexing.common.TaskToolbox;
+import org.apache.druid.indexing.common.actions.LockListAction;
+import org.apache.druid.indexing.common.actions.SurrogateAction;
+import org.apache.druid.indexing.common.actions.TaskActionClient;
+import org.apache.druid.indexing.common.task.AbstractBatchIndexTask;
+import org.apache.druid.indexing.common.task.ClientBasedTaskInfoProvider;
+import org.apache.druid.indexing.common.task.IndexTaskClientFactory;
+import org.apache.druid.indexing.common.task.TaskResource;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.RetryUtils;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.granularity.Granularity;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.segment.IndexIO;
+import org.apache.druid.segment.IndexMerger;
+import org.apache.druid.segment.IndexMergerV9;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.indexing.granularity.ArbitraryGranularitySpec;
+import org.apache.druid.segment.indexing.granularity.GranularitySpec;
+import org.apache.druid.segment.loading.DataSegmentPusher;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.HashBasedNumberedShardSpec;
+import org.apache.druid.utils.CompressionUtils;
+import org.joda.time.Interval;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * The worker task of {@link PartialSegmentMergeParallelIndexTaskRunner}. This
task reads partitioned segments created
+ * by {@link PartialSegmentGenerateTask}s, merges them, and pushes to the deep
storage. The pushed segments are reported
+ * to {@link PartialSegmentMergeParallelIndexTaskRunner}.
+ */
+public class PartialSegmentMergeTask extends AbstractBatchIndexTask
+{
+ public static final String TYPE = "partial_index_merge";
+
+ private static final Logger LOG = new Logger(PartialSegmentMergeTask.class);
+ private static final int BUFFER_SIZE = 1024 * 4;
+ private static final int NUM_FETCH_RETRIES = 3;
+
+ private final byte[] buffer = new byte[BUFFER_SIZE];
+
+ private final int numAttempts;
+ private final PartialSegmentMergeIngestionSpec ingestionSchema;
+ private final String supervisorTaskId;
+ private final IndexingServiceClient indexingServiceClient;
+ private final IndexTaskClientFactory
taskClientFactory;
+
+ @JsonCreator
+ public
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8257: Add support for parallel native indexing with shuffle for perfect rollup
clintropolis commented on a change in pull request #8257: Add support for parallel native indexing with shuffle for perfect rollup URL: https://github.com/apache/incubator-druid/pull/8257#discussion_r314201625 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AbstractBatchIndexTask.java ## @@ -360,6 +369,23 @@ private LockGranularityDetermineResult determineSegmentGranularity(ListHashBasedNumberedShardSpec always assumes that the start partitionId in the perfectly-rolled up set is 0 Should you put some of the things that it would take to support it in the javadoc just in case it does become useful to save someone in the future some trouble? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8257: Add support for parallel native indexing with shuffle for perfect rollup
clintropolis commented on a change in pull request #8257: Add support for
parallel native indexing with shuffle for perfect rollup
URL: https://github.com/apache/incubator-druid/pull/8257#discussion_r314201428
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java
##
@@ -890,80 +832,33 @@ private TaskStatus generateAndPublishSegments(
toolbox.getTaskActionClient()
.submit(SegmentTransactionalInsertAction.overwriteAction(segmentsToBeOverwritten,
segmentsToPublish));
-final Appenderator appenderator = newAppenderator(
+final Appenderator appenderator = BatchAppenderators.newAppenderator(
+getId(),
+appenderatorsManager,
buildSegmentsFireDepartmentMetrics,
toolbox,
dataSchema,
tuningConfig
);
boolean exceptionOccurred = false;
-try (
-final BatchAppenderatorDriver driver = newDriver(appenderator,
toolbox, segmentAllocator);
-final Firehose firehose =
firehoseFactory.connect(dataSchema.getParser(), firehoseTempDir)
-) {
-
+try (final BatchAppenderatorDriver driver =
BatchAppenderators.newDriver(appenderator, toolbox, segmentAllocator)) {
driver.startJob();
- while (firehose.hasMore()) {
-try {
- final InputRow inputRow = firehose.nextRow();
-
- if (inputRow == null) {
-buildSegmentsMeters.incrementThrownAway();
-continue;
- }
-
- if (!Intervals.ETERNITY.contains(inputRow.getTimestamp())) {
-final String errorMsg = StringUtils.format(
-"Encountered row with timestamp that cannot be represented as
a long: [%s]",
-inputRow
-);
-throw new ParseException(errorMsg);
- }
-
- final Optional optInterval =
granularitySpec.bucketInterval(inputRow.getTimestamp());
- if (!optInterval.isPresent()) {
-buildSegmentsMeters.incrementThrownAway();
-continue;
- }
-
- final Interval interval = optInterval.get();
- final String sequenceName =
segmentAllocator.getSequenceName(interval, inputRow);
- final AppenderatorDriverAddResult addResult = driver.add(inputRow,
sequenceName);
-
- if (addResult.isOk()) {
-
-// incremental segment publishment is allowed only when rollup
don't have to be perfect.
-if (dynamicPartitionsSpec != null) {
- final boolean isPushRequired = addResult.isPushRequired(
- dynamicPartitionsSpec.getMaxRowsPerSegment(),
- dynamicPartitionsSpec.getMaxTotalRows()
- );
- if (isPushRequired) {
-// There can be some segments waiting for being published even
though any rows won't be added to them.
-// If those segments are not published here, the available
space in appenderator will be kept to be
-// small which makes the size of segments smaller.
-final SegmentsAndMetadata pushed =
driver.pushAllAndClear(pushTimeout);
-log.info("Pushed segments[%s]", pushed.getSegments());
- }
-}
- } else {
-throw new ISE("Failed to add a row with timestamp[%s]",
inputRow.getTimestamp());
- }
-
- if (addResult.getParseException() != null) {
-handleParseException(addResult.getParseException());
- } else {
-buildSegmentsMeters.incrementProcessed();
- }
-}
-catch (ParseException e) {
- handleParseException(e);
-}
- }
-
- final SegmentsAndMetadata pushed = driver.pushAllAndClear(pushTimeout);
- log.info("Pushed segments[%s]", pushed.getSegments());
+ final FiniteFirehoseProcessor firehoseProcessor = new
FiniteFirehoseProcessor(
Review comment:
this is nice :+1:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8257: Add support for parallel native indexing with shuffle for perfect rollup
clintropolis commented on a change in pull request #8257: Add support for
parallel native indexing with shuffle for perfect rollup
URL: https://github.com/apache/incubator-druid/pull/8257#discussion_r314201396
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexTuningConfig.java
##
@@ -51,7 +52,21 @@
private final Duration chatHandlerTimeout;
private final int chatHandlerNumRetries;
- public static ParallelIndexTuningConfig defaultConfig()
+ /**
+ * Max number of segments to merge at the same time.
+ * Used only by {@link PartialSegmentMergeTask}.
+ * This configuration was temporally added to avoid using too much memory
while merging segments,
Review comment:
nit: I think maybe 'temporarily' is more appropriate word here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on issue #8305: Fix to always use end sequenceNumber for reset
gianm commented on issue #8305: Fix to always use end sequenceNumber for reset URL: https://github.com/apache/incubator-druid/pull/8305#issuecomment-521526157 Looks like Travis reported some checkstyle errors. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7514: Introduce segment limit for indexing tasks
stale[bot] commented on issue #7514: Introduce segment limit for indexing tasks URL: https://github.com/apache/incubator-druid/pull/7514#issuecomment-521571274 This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm opened a new pull request #8311: Docusaurus build framework + ingestion doc refresh.
gianm opened a new pull request #8311: Docusaurus build framework + ingestion doc refresh. URL: https://github.com/apache/incubator-druid/pull/8311 ### TL;DR A refresh of the documentation done in collaboration with @vogievetsky. Check out a render at: https://staging-druid.imply.io/docs/design/ And compare to the current doc pages: https://druid.apache.org/docs/latest/design/ ### Description This refresh has two main goals. First, setting up **Docusaurus**: 1. Use [Docusaurus](https://docusaurus.io/), a really nice documentation generator that creates much more beautiful pages than we have now. They are also more functional; each one has a left-hand collapsible table of contents showing the outline of the overall docs, and a right-hand table of contents showing the outline of that particular doc page. Compare to the old doc pages, where there's only one non-collapsible ToC, and it's grown so long that it's quite difficult to follow. 2. Make various titles and headers more consistent. The inconsistencies have been around for a while, but became much easier to notice with the new Docusaurus ToCs. Second, an **ingestion doc refresh**: 1. Includes a new `ingestion/index.md` doc that introduces all the key ingestion spec concepts, and describes the most popular ingestion methods. It is meant to be an introduction to the world of Druid ingestion. 2. Consolidate lots of disparate ingestion docs, which have grown organically over time and have become difficult to follow, into a simpler set of fewer, larger, more cross-referenced pages. Check out the new pages at: https://staging-druid.imply.io/docs/ingestion/index.html. Some prime examples of this consolidation are the new pages `ingestion/data-management.md` and `ingestion-tasks.md`, which represent multiple pages from the current set of docs. 3. Have a bit more 'opinionatedness' in the ingestion pages, pushing new people towards Kafka, Kinesis, Hadoop, and native batch ingestion. They discuss Tranquility but don't present it as something highly recommended (in keeping with its status as something that doesn't get a lot of love these days). And the few remaining references to Realtime Nodes have been redirected to a page that says they're gone now. ### WIP Work in progress; further work needed includes: 1. Restore some of the `_bin` scripts that are still useful (but which ones?) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] clintropolis commented on issue #8310: exclude kerberos extension dependencies that are already included in druid core libraries
clintropolis commented on issue #8310: exclude kerberos extension dependencies that are already included in druid core libraries URL: https://github.com/apache/incubator-druid/pull/8310#issuecomment-521559781 Oops, looks like `json-smart` can be excluded, others look like issues not specific to this extension, hopefully #8230 can resolve some of those This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] zhaomingzhu opened a new issue #8314: all tasks run failed
zhaomingzhu opened a new issue #8314: all tasks run failed
URL: https://github.com/apache/incubator-druid/issues/8314
curl bin/post-index-task --file quickstart/tutorial/deletion-index.json
--url http://localhost:8081
ask index_deletion-tutorial_2019-08-15T09:30:44.428Z still running...
Task index_deletion-tutorial_2019-08-15T09:30:44.428Z still running...
Task finished with status: FAILED
ava.util.concurrent.TimeoutException: Idle timeout expired: 30/30 ms
2019-08-15 17:07:11.239 DEBUG [qtp40936992-210]
org.eclipse.jetty.io.WriteFlusher - ignored: WriteFlusher@39210b13{IDLE}->null
java.nio.channels.ClosedChannelException
at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:532)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:353)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:216)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.io.AbstractEndPoint.shutdownInput(AbstractEndPoint.java:107)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:241)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:331)
[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:243)
[jetty-server-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
[jetty-util-9.4.10.v20180503.jar:9.4
.10.v20180503]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760)
[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678)
[jetty-util-9.4.10.v20180503.jar:9.4.10.v20180503]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
2019-08-15 17:07:11.240 DEBUG [qtp40936992-210]
org.eclipse.jetty.io.FillInterest - onClose FillInterest@5da57060{null}
2019-08-15 17:07:11.240 DEBUG [qtp40936992-210]
org.eclipse.jetty.util.thread.QueuedThreadPool - queue
org.eclipse.jetty.io.ManagedSelector$DestroyEndPo
int@3a0f8300
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] leventov commented on issue #8316: fix issue8291: make sure ReferenceCountingSegment.decrement() is invoked correctly when useCache=true
leventov commented on issue #8316: fix issue8291: make sure ReferenceCountingSegment.decrement() is invoked correctly when useCache=true URL: https://github.com/apache/incubator-druid/pull/8316#issuecomment-521699482 @pphust thank you. Please resubmit this PR against the master, we don't accept PRs into old release branches. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate
NOTICE.BINARY from NOTICE and licenses.yaml
URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314421850
##
File path: docs/_bin/generate-notice-binary.py
##
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import sys
+import yaml
+
+
+outfile = sys.stdout
+
+
+moduleHeaderLine = ""
+dependencyHeaderLine = "="
+
+def print_outfile(string):
+print(string, file=outfile)
+
+def print_error(string):
+print(string, file=sys.stderr)
+
+def print_jar(name, version, notice):
+print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, name,
version, dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(notice))
+
+def print_notice(dependency):
+if 'notice' in dependency:
+print_outfile("{} {} {} {}".format(dependencyHeaderLine,
dependency['name'], dependency['version'], dependencyHeaderLine))
+if 'libraries' in dependency:
+for library in dependency['libraries']:
+for group_id, artifact_id in library.items():
+print_outfile("{}.jar".format(artifact_id))
+print_outfile("{}\n".format(dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(dependency['notice']))
+elif 'notices' in dependency:
+for notice_entry in dependency['notices']:
+for jar, notice in notice_entry.items():
+print_jar(jar, dependency['version'], notice)
+
+
+def generate_notice(source_notice, dependences_yaml):
+# Generate NOTICE.BINARY file
Review comment:
This comment is redundant with the message string on the next line.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate
NOTICE.BINARY from NOTICE and licenses.yaml
URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314415896
##
File path: docs/_bin/generate-notice-binary.py
##
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import sys
+import yaml
+
+
+outfile = sys.stdout
+
+
+moduleHeaderLine = ""
+dependencyHeaderLine = "="
+
+def print_outfile(string):
+print(string, file=outfile)
+
+def print_error(string):
+print(string, file=sys.stderr)
+
+def print_jar(name, version, notice):
+print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, name,
version, dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(notice))
+
+def print_notice(dependency):
+if 'notice' in dependency:
+print_outfile("{} {} {} {}".format(dependencyHeaderLine,
dependency['name'], dependency['version'], dependencyHeaderLine))
+if 'libraries' in dependency:
+for library in dependency['libraries']:
+for group_id, artifact_id in library.items():
+print_outfile("{}.jar".format(artifact_id))
+print_outfile("{}\n".format(dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(dependency['notice']))
+elif 'notices' in dependency:
+for notice_entry in dependency['notices']:
+for jar, notice in notice_entry.items():
+print_jar(jar, dependency['version'], notice)
+
+
+def generate_notice(source_notice, dependences_yaml):
+# Generate NOTICE.BINARY file
+print_error("=== Generating the contents of NOTICE.BINARY file ===\n")
+
+# Print Apache license first.
+print_outfile(source_notice)
+with open(dependences_yaml) as registry_file:
+dependencies_list = list(yaml.load_all(registry_file))
+
+# Group dependencies by module
+modules_map = {}
+for dependency in dependencies_list:
+if 'notice' in dependency or 'notices' in dependency:
+if dependency['module'] not in modules_map:
Review comment:
Consider using `defaultdict` for this pattern:
https://docs.python.org/3.7/library/collections.html#defaultdict-examples
```python
from collections import defaultdict
modules_map = defaultdict(list)
for dependency in dependencies_list:
if 'notice' in dependency or 'notices' in dependency:
modules_map[dependency['module']].append(dependency)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate
NOTICE.BINARY from NOTICE and licenses.yaml
URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314415522
##
File path: docs/_bin/generate-notice-binary.py
##
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import sys
+import yaml
+
+
+outfile = sys.stdout
+
+
+moduleHeaderLine = ""
+dependencyHeaderLine = "="
+
+def print_outfile(string):
+print(string, file=outfile)
+
+def print_error(string):
+print(string, file=sys.stderr)
+
+def print_jar(name, version, notice):
+print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, name,
version, dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(notice))
+
+def print_notice(dependency):
+if 'notice' in dependency:
+print_outfile("{} {} {} {}".format(dependencyHeaderLine,
dependency['name'], dependency['version'], dependencyHeaderLine))
+if 'libraries' in dependency:
+for library in dependency['libraries']:
+for group_id, artifact_id in library.items():
+print_outfile("{}.jar".format(artifact_id))
+print_outfile("{}\n".format(dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(dependency['notice']))
+elif 'notices' in dependency:
+for notice_entry in dependency['notices']:
+for jar, notice in notice_entry.items():
+print_jar(jar, dependency['version'], notice)
+
+
+def generate_notice(source_notice, dependences_yaml):
+# Generate NOTICE.BINARY file
+print_error("=== Generating the contents of NOTICE.BINARY file ===\n")
+
+# Print Apache license first.
+print_outfile(source_notice)
+with open(dependences_yaml) as registry_file:
+dependencies_list = list(yaml.load_all(registry_file))
Review comment:
Does it make sense to use a `set` instead of a `list`? Using a `set` may
also be beneficial for the membership test below (`O(1)` versus `O(n)`).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314411868 ## File path: docs/_bin/generate-notice-binary.py ## @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import sys +import yaml + + +outfile = sys.stdout + + +moduleHeaderLine = "" Review comment: FYI, you can do something like: `moduleHeaderLine = "#" * 12` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314422995 ## File path: docs/_bin/generate-notice-binary.py ## @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import sys +import yaml + + +outfile = sys.stdout + + +moduleHeaderLine = "" +dependencyHeaderLine = "=" + +def print_outfile(string): +print(string, file=outfile) + +def print_error(string): Review comment: This name is a bit misleading as it's not used to print informational messages instead of error messages This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate
NOTICE.BINARY from NOTICE and licenses.yaml
URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314421079
##
File path: docs/_bin/generate-notice-binary.py
##
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import sys
+import yaml
+
+
+outfile = sys.stdout
+
+
+moduleHeaderLine = ""
+dependencyHeaderLine = "="
+
+def print_outfile(string):
+print(string, file=outfile)
+
+def print_error(string):
+print(string, file=sys.stderr)
+
+def print_jar(name, version, notice):
+print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, name,
version, dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(notice))
+
+def print_notice(dependency):
+if 'notice' in dependency:
+print_outfile("{} {} {} {}".format(dependencyHeaderLine,
dependency['name'], dependency['version'], dependencyHeaderLine))
+if 'libraries' in dependency:
+for library in dependency['libraries']:
+for group_id, artifact_id in library.items():
+print_outfile("{}.jar".format(artifact_id))
+print_outfile("{}\n".format(dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(dependency['notice']))
+elif 'notices' in dependency:
+for notice_entry in dependency['notices']:
+for jar, notice in notice_entry.items():
+print_jar(jar, dependency['version'], notice)
+
+
+def generate_notice(source_notice, dependences_yaml):
+# Generate NOTICE.BINARY file
+print_error("=== Generating the contents of NOTICE.BINARY file ===\n")
+
+# Print Apache license first.
+print_outfile(source_notice)
+with open(dependences_yaml) as registry_file:
+dependencies_list = list(yaml.load_all(registry_file))
+
+# Group dependencies by module
+modules_map = {}
+for dependency in dependencies_list:
+if 'notice' in dependency or 'notices' in dependency:
+if dependency['module'] not in modules_map:
+modules_map[dependency['module']] = []
+module_dependencies = modules_map[dependency['module']]
+module_dependencies.append(dependency)
+
+for module_name, dependencies_of_module in modules_map.items():
+print_outfile("{} BINARY/{} {}\n".format(moduleHeaderLine,
module_name.upper(), moduleHeaderLine))
+for dependency in dependencies_of_module:
+print_notice(dependency)
+
+
+if __name__ == "__main__":
+try:
+parser = argparse.ArgumentParser(description='generate binary notice
file.')
+parser.add_argument('notice', metavar='',
type=str)
+parser.add_argument('license_yaml', metavar='',
type=str)
+parser.add_argument('out_path', metavar='',
type=str)
+args = parser.parse_args()
+
+with open(args.notice) as apache_notice_file:
+source_notice = apache_notice_file.read()
+dependencies_yaml = args.license_yaml
+
+with open(args.out_path, "w") as outfile:
Review comment:
FYI, if it's desired for this to fail if the file already exists, then the
"x" mode can be used.
Also, this line has the side-effect of changing the behavior of
`print_outfile()` which is a bit hidden as it goes through the `outfile` global
variable.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314411276 ## File path: docs/_bin/generate-notice-binary.py ## @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import sys +import yaml + + +outfile = sys.stdout Review comment: Typically, constants are in UPPER_SNAKE_CASE: https://www.python.org/dev/peps/pep-0008/#constants When is the value `sys.stdout` used? `out_path` is a required positional command-line arg, so this variable should always get initialized to that value. If `sys.stdout` is not used there, then perhaps `print_error` should be printing to stdout instead? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate
NOTICE.BINARY from NOTICE and licenses.yaml
URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314412918
##
File path: docs/_bin/generate-notice-binary.py
##
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import sys
+import yaml
+
+
+outfile = sys.stdout
+
+
+moduleHeaderLine = ""
+dependencyHeaderLine = "="
+
+def print_outfile(string):
+print(string, file=outfile)
+
+def print_error(string):
+print(string, file=sys.stderr)
+
+def print_jar(name, version, notice):
+print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, name,
version, dependencyHeaderLine))
Review comment:
Consider using an f-string instead of `string.format()`:
https://www.python.org/dev/peps/pep-0498/#code-equivalence
```python
print_outfile(f"{dependencyHeaderLine} {name}-{version}.jar
{dependencyHeaderLine}")
```
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] ccaominh commented on a change in pull request #8306: autogenerate NOTICE.BINARY from NOTICE and licenses.yaml
ccaominh commented on a change in pull request #8306: autogenerate
NOTICE.BINARY from NOTICE and licenses.yaml
URL: https://github.com/apache/incubator-druid/pull/8306#discussion_r314418072
##
File path: docs/_bin/generate-notice-binary.py
##
@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import sys
+import yaml
+
+
+outfile = sys.stdout
+
+
+moduleHeaderLine = ""
+dependencyHeaderLine = "="
+
+def print_outfile(string):
+print(string, file=outfile)
+
+def print_error(string):
+print(string, file=sys.stderr)
+
+def print_jar(name, version, notice):
+print_outfile("{} {}-{}.jar {}".format(dependencyHeaderLine, name,
version, dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(notice))
+
+def print_notice(dependency):
+if 'notice' in dependency:
+print_outfile("{} {} {} {}".format(dependencyHeaderLine,
dependency['name'], dependency['version'], dependencyHeaderLine))
+if 'libraries' in dependency:
+for library in dependency['libraries']:
+for group_id, artifact_id in library.items():
+print_outfile("{}.jar".format(artifact_id))
+print_outfile("{}\n".format(dependencyHeaderLine))
+print_outfile("{}\n\n\n\n".format(dependency['notice']))
+elif 'notices' in dependency:
+for notice_entry in dependency['notices']:
+for jar, notice in notice_entry.items():
+print_jar(jar, dependency['version'], notice)
+
+
+def generate_notice(source_notice, dependences_yaml):
+# Generate NOTICE.BINARY file
+print_error("=== Generating the contents of NOTICE.BINARY file ===\n")
+
+# Print Apache license first.
+print_outfile(source_notice)
+with open(dependences_yaml) as registry_file:
+dependencies_list = list(yaml.load_all(registry_file))
+
+# Group dependencies by module
+modules_map = {}
+for dependency in dependencies_list:
+if 'notice' in dependency or 'notices' in dependency:
+if dependency['module'] not in modules_map:
+modules_map[dependency['module']] = []
+module_dependencies = modules_map[dependency['module']]
+module_dependencies.append(dependency)
+
+for module_name, dependencies_of_module in modules_map.items():
+print_outfile("{} BINARY/{} {}\n".format(moduleHeaderLine,
module_name.upper(), moduleHeaderLine))
+for dependency in dependencies_of_module:
+print_notice(dependency)
+
+
+if __name__ == "__main__":
+try:
+parser = argparse.ArgumentParser(description='generate binary notice
file.')
+parser.add_argument('notice', metavar='',
type=str)
+parser.add_argument('license_yaml', metavar='',
type=str)
+parser.add_argument('out_path', metavar='',
type=str)
+args = parser.parse_args()
+
+with open(args.notice) as apache_notice_file:
+source_notice = apache_notice_file.read()
+dependencies_yaml = args.license_yaml
+
+with open(args.out_path, "w") as outfile:
+generate_notice(source_notice, dependencies_yaml)
+
+except KeyboardInterrupt:
+print('Interrupted, closing.')
Review comment:
I'm guessing the weird character showing up here is because of a missing
newline at the end of the file?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson opened a new issue #8318: Rename `maxNumSubTasks` to `maxNumConcurrentSubTasks` for native parallel index task
jihoonson opened a new issue #8318: Rename `maxNumSubTasks` to `maxNumConcurrentSubTasks` for native parallel index task URL: https://github.com/apache/incubator-druid/issues/8318 ### Description `maxNumSubTasks` actually means the number of max sub tasks that can be run concurrently at the same time. It's very confusing and should be renamed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8257: Add support for parallel native indexing with shuffle for perfect rollup
jihoonson commented on a change in pull request #8257: Add support for parallel
native indexing with shuffle for perfect rollup
URL: https://github.com/apache/incubator-druid/pull/8257#discussion_r314472836
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexTuningConfig.java
##
@@ -43,6 +42,8 @@
private static final Duration DEFAULT_CHAT_HANDLER_TIMEOUT = new
Period("PT10S").toStandardDuration();
private static final int DEFAULT_CHAT_HANDLER_NUM_RETRIES = 5;
+ private static final int DEFAULT_MAX_NUM_SEGMENTS_TO_MERGE = 100;
+ private static final int DEFAULT_TOTAL_NUM_MERGE_TASKS = 10;
Review comment:
`maxNumSubTasks` is actually the number of max number of sub tasks that can
be run concurrently at the same time. Raised
https://github.com/apache/incubator-druid/issues/8318.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug opened a new pull request #8319: string column handling for long/float min/max/sum aggregators
himanshug opened a new pull request #8319: string column handling for long/float min/max/sum aggregators URL: https://github.com/apache/incubator-druid/pull/8319 Fixes #8148 Same as #8243 but for long/float min/max/sum aggregators ### Description This patch adds handling of single/multi value column handling by long/float sum/min/max aggregators to do a best effort parsing string as long/float. `StringColumn[Long/Float]AggregatorWrapper` and `StringColumn[Long/Float]BufferAggregatorWrapper` classes are introduced that can wrap existing long/float aggregators to handle string columns. Both of the classes are used by `Simple[Long/Float]AggregatorFactory` to be used when input column is known to be of String type. This PR has: - [x] been self-reviewed. - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [x] added unit tests or modified existing tests to cover new code paths. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7319: Fix unrealistic test variables in KafkaSupervisorTest and tidy up unused variable in checkpointing process
jihoonson commented on a change in pull request #7319: Fix unrealistic test variables in KafkaSupervisorTest and tidy up unused variable in checkpointing process URL: https://github.com/apache/incubator-druid/pull/7319#discussion_r314510129 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java ## @@ -338,19 +339,16 @@ public void handle() @Deprecated private final String baseSequenceName; private final SeekableStreamDataSourceMetadata previousCheckpoint; -private final SeekableStreamDataSourceMetadata currentCheckpoint; -public CheckpointNotice( +CheckpointNotice( @Nullable Integer nullableTaskGroupId, @Deprecated String baseSequenceName, -SeekableStreamDataSourceMetadata previousCheckpoint, -SeekableStreamDataSourceMetadata currentCheckpoint +SeekableStreamDataSourceMetadata previousCheckpoint Review comment: Thanks, fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid-website] clintropolis opened a new pull request #29: update website for 0.15.1-incubating
clintropolis opened a new pull request #29: update website for 0.15.1-incubating URL: https://github.com/apache/incubator-druid-website/pull/29 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website-src] branch 0.15.1-incubating updated (2faf8cd -> ad2ed1c)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch 0.15.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git. discard 2faf8cd update website for 0.15.1-incubating add ad2ed1c fix it This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (2faf8cd) \ N -- N -- N refs/heads/0.15.1-incubating (ad2ed1c) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: _config.yml| 4 +- docs/0.14.0-incubating/configuration/index.md | 28 +- docs/0.14.0-incubating/configuration/logging.md| 33 -- docs/0.14.0-incubating/configuration/realtime.md | 2 +- .../dependencies/metadata-storage.md | 5 +- docs/0.14.0-incubating/design/coordinator.md | 3 +- docs/0.14.0-incubating/design/index.md | 41 +-- docs/0.14.0-incubating/design/segments.md | 2 +- docs/0.14.0-incubating/development/build.md| 5 +- docs/0.14.0-incubating/development/experimental.md | 19 +- .../extensions-contrib/distinctcount.md| 4 +- .../development/extensions-contrib/influx.md | 2 - .../extensions-contrib/materialized-view.md| 3 - .../extensions-core/druid-basic-security.md| 151 +--- .../development/extensions-core/druid-lookups.md | 1 - .../development/extensions-core/kafka-ingestion.md | 3 +- .../development/extensions-core/parquet.md | 9 +- .../development/extensions-core/s3.md | 10 +- docs/0.14.0-incubating/development/extensions.md | 13 +- docs/0.14.0-incubating/development/modules.md | 6 +- docs/0.14.0-incubating/development/overview.md | 2 +- docs/0.14.0-incubating/development/router.md | 5 - docs/0.14.0-incubating/ingestion/firehose.md | 64 +--- .../ingestion/hadoop-vs-native-batch.md| 4 +- docs/0.14.0-incubating/ingestion/hadoop.md | 2 +- docs/0.14.0-incubating/ingestion/index.md | 2 +- docs/0.14.0-incubating/ingestion/ingestion-spec.md | 6 +- docs/0.14.0-incubating/ingestion/native_tasks.md | 11 +- docs/0.14.0-incubating/misc/math-expr.md | 16 +- docs/0.14.0-incubating/operations/api-reference.md | 67 ++-- docs/0.14.0-incubating/operations/druid-console.md | 51 +-- .../operations/img/01-home-view.png| Bin 58587 -> 60287 bytes .../operations/insert-segment-to-db.md | 153 ++-- docs/0.14.0-incubating/operations/pull-deps.md | 14 +- .../operations/recommendations.md | 8 +- .../operations/rule-configuration.md | 2 + docs/0.14.0-incubating/operations/tls-support.md | 15 +- docs/0.14.0-incubating/querying/aggregations.md| 29 +- docs/0.14.0-incubating/querying/caching.md | 44 +-- docs/0.14.0-incubating/querying/filters.md | 6 - docs/0.14.0-incubating/querying/groupbyquery.md| 3 +- docs/0.14.0-incubating/querying/lookups.md | 13 +- docs/0.14.0-incubating/querying/multitenancy.md| 2 +- docs/0.14.0-incubating/querying/query-context.md | 4 +- docs/0.14.0-incubating/querying/querying.md| 36 +- docs/0.14.0-incubating/querying/scan-query.md | 120 +++ docs/0.14.0-incubating/querying/select-query.md| 17 +- docs/0.14.0-incubating/querying/sql.md | 159 - docs/0.14.0-incubating/toc.md | 173 + docs/0.14.0-incubating/tutorials/cluster.md| 390 - .../tutorials/img/tutorial-deletion-02.png | Bin 810422 -> 200459 bytes docs/0.14.0-incubating/tutorials/index.md | 116 +++--- .../tutorials/tutorial-batch-hadoop.md | 20 +- docs/0.14.0-incubating/tutorials/tutorial-batch.md | 154 ++-- .../tutorials/tutorial-compaction.md | 9 +- .../tutorials/tutorial-delete-data.md | 60 ++-- .../tutorials/tutorial-ingestion-spec.md | 4 +- docs/0.14.0-incubating/tutorials/tutorial-kafka.md | 103 +- docs/0.14.0-incubating/tutorials/tutorial-query.md | 329 + .../tutorials/tutorial-retention.md| 2 +- .../0.14.0-incubating/tutorials/tutorial-rollup.md | 7 +- .../tutorials/tutorial-tranquility.md | 14 +-
[incubator-druid-website-src] 01/01: Merge pull request #37 from apache/0.15.1-incubating
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git commit 3b2ac46c1b705284b5940950a977985befe2f257 Merge: f2509a8 68edf87 Author: Fangjin Yang AuthorDate: Thu Aug 15 16:24:01 2019 -0700 Merge pull request #37 from apache/0.15.1-incubating update website for 0.15.1-incubating .../extensions-contrib/distinctcount.md| 4 +- .../development/extensions-contrib/influx.md | 2 + .../extensions-contrib/materialized-view.md| 3 + .../extensions-contrib/momentsketch-quantiles.md | 6 + .../extensions-contrib/moving-average-query.md | 9 ++ .../extensions-core/druid-basic-security.md| 137 - .../development/extensions-core/druid-lookups.md | 1 + docs/latest/development/extensions-core/orc.md | 4 + docs/latest/development/extensions.md | 3 +- docs/latest/operations/pull-deps.md| 14 +-- docs/latest/querying/filters.md| 6 + docs/latest/tutorials/cluster.md | 10 +- docs/latest/tutorials/index.md | 26 ++-- docs/latest/tutorials/tutorial-batch-hadoop.md | 6 +- docs/latest/tutorials/tutorial-batch.md| 2 +- docs/latest/tutorials/tutorial-ingestion-spec.md | 2 +- docs/latest/tutorials/tutorial-rollup.md | 2 +- docs/latest/tutorials/tutorial-tranquility.md | 6 +- 18 files changed, 205 insertions(+), 38 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website-src] branch 0.15.1-incubating deleted (was 68edf87)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch 0.15.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git. was 68edf87 0.15.1-incubating The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid-website-src] fjy merged pull request #37: update website for 0.15.1-incubating
fjy merged pull request #37: update website for 0.15.1-incubating URL: https://github.com/apache/incubator-druid-website-src/pull/37 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website-src] branch master updated (f2509a8 -> 3b2ac46)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git. from f2509a8 Merge pull request #34 from rkingsland/patch-9 add 68edf87 0.15.1-incubating new 3b2ac46 Merge pull request #37 from apache/0.15.1-incubating The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../extensions-contrib/distinctcount.md| 4 +- .../development/extensions-contrib/influx.md | 2 + .../extensions-contrib/materialized-view.md| 3 + .../extensions-contrib/momentsketch-quantiles.md | 6 + .../extensions-contrib/moving-average-query.md | 9 ++ .../extensions-core/druid-basic-security.md| 137 - .../development/extensions-core/druid-lookups.md | 1 + docs/latest/development/extensions-core/orc.md | 4 + docs/latest/development/extensions.md | 3 +- docs/latest/operations/pull-deps.md| 14 +-- docs/latest/querying/filters.md| 6 + docs/latest/tutorials/cluster.md | 10 +- docs/latest/tutorials/index.md | 26 ++-- docs/latest/tutorials/tutorial-batch-hadoop.md | 6 +- docs/latest/tutorials/tutorial-batch.md| 2 +- docs/latest/tutorials/tutorial-ingestion-spec.md | 2 +- docs/latest/tutorials/tutorial-rollup.md | 2 +- docs/latest/tutorials/tutorial-tranquility.md | 6 +- 18 files changed, 205 insertions(+), 38 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website-src] branch master updated: Update events.yml
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git The following commit(s) were added to refs/heads/master by this push: new aa3d5ff Update events.yml new 3971427 Merge pull request #36 from rkingsland/patch-11 aa3d5ff is described below commit aa3d5ffd273e1ccf3e18edda75f88c34d887ece6 Author: rkingsland AuthorDate: Thu Aug 15 13:09:53 2019 -0700 Update events.yml --- _data/events.yml | 21 ++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/_data/events.yml b/_data/events.yml index e7500af..919cd6f 100644 --- a/_data/events.yml +++ b/_data/events.yml @@ -1,14 +1,29 @@ - date: 2019-08-28 - name: "SF Big Analytics Meetup - Apache Druid and YuniKorn: Universal Resource scheduler for both K8s and Yarn" + name: "Meetup: SF Big Analytics - Apache Druid and YuniKorn: Universal Resource scheduler for both K8s and Yarn" info: "2019-08-28, from 6:00 - 9:00 PM @ Workday, 160 Spear St Suite 1700 · San Francisco, CA" link: https://www.meetup.com/SF-Big-Analytics/events/263274680/ + +- date: 2019-08-21 + name: "Meetup: Utah Data Engineering - Two Talks About Real Time w/ Jon King (Imply) and Thor Sigurjonsson (Overstock) + info: "2019-08-21, from 6:30 - 8:30 PM @ Overstock, 799 Coliseum Way · Midvale, UT" + link: https://www.meetup.com/utah-data-engineering-meetup/events/263761070/ - date: 2019-09-09 name: "ApacheCon North America, Las Vegas" - info: "2019-09-09 to 2019-09-13 Flamingo Las Vegas, 3555 S Las Vegas Blvd, Las Vegas, NV 89109, USA" + info: "2019-09-09 to 2019-09-13, Flamingo Las Vegas, 3555 S Las Vegas Blvd, Las Vegas, NV 89109, USA" link: https://apachecon.com/acna19/ + +- date: 2019-09-17 + name: "Meetup: Denver Apache Druid Meetup by Imply - Real-time Analytics with Apache Druid at Fullcontact" + info: "2019-09-17, from 6:00 - 9:00 PM @ FullContact, 1200 17th St · Denver, CO. 13th Floor + link: https://www.meetup.com/Denver-Apache-Druid-Meetup-by-Imply/events/264007236/ + +- date: 2019-09-19 + name: "Meetup: Big Data and Cloud - Rethinking Druid's user experience" + info: "2019-09-19, from 6:00 - 8:30 PM @ Hacker Dojo, 3350 Thomas Road · Santa Clara, CA" + link: https://www.meetup.com/datariders/events/263980371/ - date: 2019-10-22 name: "ApacheCon Europe, Berlin" - info: "2019-10-22 to 2019-10-25 Kulturbrauerei, Schönhauser Allee 36, 10435 Berlin, Germany" + info: "2019-10-22 to 2019-10-25, Kulturbrauerei, Schönhauser Allee 36, 10435 Berlin, Germany" link: https://aceu19.apachecon.com/ - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid-website-src] fjy merged pull request #36: Update events.yml
fjy merged pull request #36: Update events.yml URL: https://github.com/apache/incubator-druid-website-src/pull/36 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website] 01/01: Merge pull request #29 from apache/0.15.1-incubating
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git commit 185560d7ef355250d866acf80cad68f2b00823ae Merge: 37b1ea4 9f91b15 Author: Jihoon Son AuthorDate: Thu Aug 15 16:43:27 2019 -0700 Merge pull request #29 from apache/0.15.1-incubating update website for 0.15.1-incubating .../extensions-contrib/distinctcount.html | 6 +- .../development/extensions-contrib/influx.html | 53 +-- .../extensions-contrib/materialized-view.html | 135 --- .../extensions-contrib/momentsketch-quantiles.html | 133 --- .../extensions-contrib/moving-average-query.html | 395 ++--- .../extensions-core/druid-basic-security.html | 126 ++- .../development/extensions-core/druid-lookups.html | 33 +- docs/latest/development/extensions-core/orc.html | 80 +++-- docs/latest/development/extensions.html| 7 +- docs/latest/operations/pull-deps.html | 14 +- docs/latest/querying/filters.html | 122 +++ docs/latest/tutorials/cluster.html | 10 +- docs/latest/tutorials/index.html | 26 +- docs/latest/tutorials/tutorial-batch-hadoop.html | 6 +- docs/latest/tutorials/tutorial-batch.html | 2 +- docs/latest/tutorials/tutorial-ingestion-spec.html | 2 +- docs/latest/tutorials/tutorial-rollup.html | 2 +- docs/latest/tutorials/tutorial-tranquility.html| 6 +- druid-powered.html | 5 + 19 files changed, 637 insertions(+), 526 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website] branch asf-site updated (37b1ea4 -> 185560d)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git. from 37b1ea4 Merge pull request #28 from implydata/autobuild add 9f91b15 0.15.1 incubating new 185560d Merge pull request #29 from apache/0.15.1-incubating The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../extensions-contrib/distinctcount.html | 6 +- .../development/extensions-contrib/influx.html | 53 +-- .../extensions-contrib/materialized-view.html | 135 --- .../extensions-contrib/momentsketch-quantiles.html | 133 --- .../extensions-contrib/moving-average-query.html | 395 ++--- .../extensions-core/druid-basic-security.html | 126 ++- .../development/extensions-core/druid-lookups.html | 33 +- docs/latest/development/extensions-core/orc.html | 80 +++-- docs/latest/development/extensions.html| 7 +- docs/latest/operations/pull-deps.html | 14 +- docs/latest/querying/filters.html | 122 +++ docs/latest/tutorials/cluster.html | 10 +- docs/latest/tutorials/index.html | 26 +- docs/latest/tutorials/tutorial-batch-hadoop.html | 6 +- docs/latest/tutorials/tutorial-batch.html | 2 +- docs/latest/tutorials/tutorial-ingestion-spec.html | 2 +- docs/latest/tutorials/tutorial-rollup.html | 2 +- docs/latest/tutorials/tutorial-tranquility.html| 6 +- druid-powered.html | 5 + 19 files changed, 637 insertions(+), 526 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid-website] jihoonson merged pull request #29: update website for 0.15.1-incubating
jihoonson merged pull request #29: update website for 0.15.1-incubating URL: https://github.com/apache/incubator-druid-website/pull/29 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website] branch 0.15.1-incubating deleted (was 9f91b15)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch 0.15.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git. was 9f91b15 0.15.1 incubating The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jon-wei commented on a change in pull request #8287: Cleanup appenderators and
segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314546755
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -134,41 +154,30 @@ public Appenderator createRealtimeAppenderatorForTask(
CachePopulatorStats cachePopulatorStats
)
{
-SinkQuerySegmentWalker segmentWalker =
datasourceSegmentWalkers.computeIfAbsent(
-schema.getDataSource(),
-(datasource) -> {
- VersionedIntervalTimeline sinkTimeline = new
VersionedIntervalTimeline<>(
- String.CASE_INSENSITIVE_ORDER
- );
- SinkQuerySegmentWalker datasourceSegmentWalker = new
SinkQuerySegmentWalker(
- schema.getDataSource(),
- sinkTimeline,
- objectMapper,
- emitter,
- conglomerate,
- this.queryExecutorService,
- Preconditions.checkNotNull(this.cache, "cache"),
- this.cacheConfig,
- this.cachePopulatorStats
- );
- return datasourceSegmentWalker;
-}
-);
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.computeIfAbsent(
+ schema.getDataSource(),
+ (datasource) -> {
+return new DatasourceBundle(datasource);
Review comment:
Changed to `DatasourceBundle::new`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jon-wei commented on a change in pull request #8287: Cleanup appenderators and
segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314546703
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -248,6 +289,50 @@ private AppenderatorConfig
rewriteAppenderatorConfigMemoryLimits(AppenderatorCon
return new MemoryParameterOverridingAppenderatorConfig(baseConfig,
perWorkerLimit);
}
+ private class DatasourceBundle
+ {
+private final SinkQuerySegmentWalker walker;
+private final Map> taskAppenderatorMap;
+
+public DatasourceBundle(
Review comment:
Hm, I made the class public and added a `@VisibleForTesting` annotation,
since it's accessed from `TaskLifecycleTest` now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jon-wei commented on a change in pull request #8287: Cleanup appenderators and
segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314547644
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -248,6 +289,50 @@ private AppenderatorConfig
rewriteAppenderatorConfigMemoryLimits(AppenderatorCon
return new MemoryParameterOverridingAppenderatorConfig(baseConfig,
perWorkerLimit);
}
+ private class DatasourceBundle
+ {
+private final SinkQuerySegmentWalker walker;
+private final Map> taskAppenderatorMap;
+
+public DatasourceBundle(
+String dataSource
+)
+{
+ this.taskAppenderatorMap = new HashMap<>();
+
+ VersionedIntervalTimeline sinkTimeline = new
VersionedIntervalTimeline<>(
+ String.CASE_INSENSITIVE_ORDER
+ );
+ this.walker = new SinkQuerySegmentWalker(
+ dataSource,
+ sinkTimeline,
+ objectMapper,
+ serviceEmitter,
+ queryRunnerFactoryConglomerateProvider.get(),
+ queryExecutorService,
+ Preconditions.checkNotNull(cache, "cache"),
+ cacheConfig,
+ cachePopulatorStats
+ );
+}
+
+public SinkQuerySegmentWalker getWalker()
+{
+ return walker;
+}
+
+public void addAppenderator(String taskId, Appenderator appenderator)
+{
+ taskAppenderatorMap.computeIfAbsent(
+ taskId,
+ (myTaskId) -> {
+return new ArrayList<>();
+ }
+ );
+ taskAppenderatorMap.get(taskId).add(appenderator);
Review comment:
Changed, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website] branch fix-0.15.1-downloads created (now 537e271)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch fix-0.15.1-downloads in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git. at 537e271 fix 0.15.1 download links This branch includes the following new commits: new 537e271 fix 0.15.1 download links The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jon-wei commented on a change in pull request #8287: Cleanup appenderators and
segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314547560
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -183,22 +192,48 @@ public Appenderator createOfflineAppenderatorForTask(
IndexMerger indexMerger
)
{
-Appenderator appenderator = Appenderators.createOffline(
-schema,
-rewriteAppenderatorConfigMemoryLimits(config),
-metrics,
-dataSegmentPusher,
-objectMapper,
-indexIO,
-wrapIndexMerger(indexMerger)
-);
-return appenderator;
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.computeIfAbsent(
+ schema.getDataSource(),
+ (datasource) -> {
+return new DatasourceBundle(datasource);
+ }
+ );
+
+ Appenderator appenderator = Appenderators.createOffline(
+ schema,
+ rewriteAppenderatorConfigMemoryLimits(config),
+ metrics,
+ dataSegmentPusher,
+ objectMapper,
+ indexIO,
+ wrapIndexMerger(indexMerger)
+ );
+ datasourceBundle.addAppenderator(taskId, appenderator);
+ return appenderator;
+}
}
@Override
- public void removeAppenderatorForTask(String taskId)
+ public void removeAppenderatorsForTask(
+ String taskId,
+ String dataSource
+ )
{
-// nothing to remove presently
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.get(dataSource);
+ if (datasourceBundle == null) {
+LOG.error("Could not find datasource bundle for [%s], task [%s]",
dataSource, taskId);
Review comment:
I think this is more of an error at the level of the task manager and it
would only be called after a task has finished (so I don't think it could be
propagated to tasks), it's not fatal enough to stop altogether but I think it
indicates bugs so I wanted to be ERROR instead of WARN.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid-website-src] clintropolis opened a new pull request #38: fix versions, events
clintropolis opened a new pull request #38: fix versions, events URL: https://github.com/apache/incubator-druid-website-src/pull/38 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #7363: Implementing dropwizard emitter for druid
jihoonson commented on issue #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#issuecomment-521814221 @nishantmonu51 sorry for the late response. Will take a look sometime this week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314533828
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitter.java
##
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.core.Emitter;
+import org.apache.druid.java.util.emitter.core.Event;
+import org.apache.druid.java.util.emitter.service.AlertEvent;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class DropwizardEmitter implements Emitter
+{
+ private static Logger log = new Logger(DropwizardEmitter.class);
+ private final MetricRegistry metricsRegistry = new MetricRegistry();
+ private final AtomicBoolean started = new AtomicBoolean(false);
+ private final DropwizardConverter converter;
+ private final List alertEmitters;
+ private final List reporters;
+ private final DropwizardEmitterConfig config;
+ // Note: the gauges do not represent the actual instantaneous value for the
metrics.
+ // Instead they have the last known value for it.
+ private final ConcurrentHashMap gagues = new
ConcurrentHashMap<>();
+
+ public DropwizardEmitter(
+ DropwizardEmitterConfig config,
+ ObjectMapper mapper,
+ List alertEmitters
+ )
+ {
+this.alertEmitters = alertEmitters;
+this.config = config;
+this.reporters = config.getReporters();
+this.converter = new DropwizardConverter(mapper,
config.getDimensionMapPath());
+ }
+
+
+ @Override
+ public void start()
Review comment:
Hmm, when can it be called twice? It should be managed by the lifecycle.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314540257
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardConsoleReporter.java
##
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard.reporters;
+
+import com.codahale.metrics.ConsoleReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.emitter.dropwizard.DropwizardReporter;
+
+import java.util.concurrent.TimeUnit;
+
+@JsonTypeName("console")
Review comment:
This can be removed since it's already registered in `DropwizardReporter`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314535224
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitter.java
##
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.codahale.metrics.Gauge;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.emitter.core.Emitter;
+import org.apache.druid.java.util.emitter.core.Event;
+import org.apache.druid.java.util.emitter.service.AlertEvent;
+import org.apache.druid.java.util.emitter.service.ServiceMetricEvent;
+
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+public class DropwizardEmitter implements Emitter
+{
+ private static final Logger log = new Logger(DropwizardEmitter.class);
+ private final MetricRegistry metricsRegistry = new MetricRegistry();
+ private final AtomicBoolean started = new AtomicBoolean(false);
+ private final DropwizardConverter converter;
+ private final List alertEmitters;
+ private final List reporters;
+ private final DropwizardEmitterConfig config;
+ // Note: the gauges do not represent the actual instantaneous value for the
metrics.
+ // Instead they have the last known value for it.
+ private final Map gagues;
+
+ public DropwizardEmitter(
+ DropwizardEmitterConfig config,
+ ObjectMapper mapper,
+ List alertEmitters
+ )
+ {
+this.alertEmitters = alertEmitters;
+this.config = config;
+this.reporters = config.getReporters();
+this.converter = new DropwizardConverter(mapper,
config.getDimensionMapPath());
+this.gagues = Collections.synchronizedMap(new
GaugesCache<>(config.getMaxGaugeCount()));
+ }
+
+
+ @Override
+ public void start()
+ {
+final boolean alreadyStarted = started.getAndSet(true);
+if (!alreadyStarted) {
+ for (DropwizardReporter reporter : reporters) {
+reporter.start(metricsRegistry);
+ }
+}
+ }
+
+ @Override
+ public void emit(Event event)
+ {
+synchronized (started) {
+ if (!started.get()) {
+throw new RejectedExecutionException("Dropwizard emitter Service not
started.");
+ }
+}
+if (event instanceof ServiceMetricEvent) {
+ ServiceMetricEvent metricEvent = (ServiceMetricEvent) event;
+ String host = metricEvent.getHost();
+ String service = metricEvent.getService();
+ String metric = metricEvent.getMetric();
+ Map userDims = metricEvent.getUserDims();
+ Number value = metricEvent.getValue();
+ ImmutableList.Builder nameBuilder = new
ImmutableList.Builder<>();
+ LinkedHashMap dims = new LinkedHashMap<>();
+ final DropwizardMetricSpec metricSpec =
converter.addFilteredUserDims(service, metric, userDims, dims);
+
+ if (metricSpec != null) {
+if (config.getPrefix() != null) {
+ nameBuilder.add(config.getPrefix());
+}
+nameBuilder.add("metric=" + metric);
+nameBuilder.add("service=" + service);
+if (config.getIncludeHost()) {
+ nameBuilder.add("hostname=" + host);
+}
+dims.forEach((key, value1) -> nameBuilder.add(key + "=" + value1));
+
+String fullName =
StringUtils.replaceChar(Joiner.on(",").join(nameBuilder.build()), '/', ".");
+updateMetric(fullName, value, metricSpec);
+ } else {
+log.debug("Service=[%s], Metric=[%s] has no mapping", service, metric);
+ }
+} else if (event instanceof AlertEvent) {
+ for (Emitter emitter : alertEmitters) {
+emitter.emit(event);
+ }
+} else {
+
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314531705
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardConverter.java
##
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Strings;
+import org.apache.curator.shaded.com.google.common.io.Closeables;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+/**
+ *
+ */
+public class DropwizardConverter
+{
+ private static final Logger log = new Logger(DropwizardConverter.class);
+ private final Map metricMap;
+
+ public DropwizardConverter(ObjectMapper mapper, String dimensionMapPath)
+ {
+metricMap = readMap(mapper, dimensionMapPath);
+ }
+
+ /**
+ * Filters user dimensions for given metric and adds them to
filteredDimensions.
+ * Returns null if there is no mapping present for the given metric.
+ */
+ @Nullable
+ public DropwizardMetricSpec addFilteredUserDims(
+ String service,
+ String metric,
+ Map userDims,
+ Map filteredDimensions
+ )
+ {
+ /*
+Find the metric in the map. If we cant find it try to look it up
prefixed by the service name.
Review comment:
nit: I think multi line comments are not widely used. Perhaps better to use
the same style.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314529356 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| +|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the dropwizard metric type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge metrics to be cached at any time. |no| 100K| + + +### Druid to Dropwizard Event Conversion + +Each metric emitted using Dropwizard must specify a type, one of `[timer, counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to +be provided as a JSON file. Additionally, this mapping specifies which dimensions should be included for each metric. +If the user does not specify their own JSON file, a [default mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json) is used. Review comment: I think it would be better to add the whole metrics here rather than pointing this file since the file can be changed between releases. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jon-wei commented on a change in pull request #8287: Cleanup appenderators and
segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314546982
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -183,22 +192,48 @@ public Appenderator createOfflineAppenderatorForTask(
IndexMerger indexMerger
)
{
-Appenderator appenderator = Appenderators.createOffline(
-schema,
-rewriteAppenderatorConfigMemoryLimits(config),
-metrics,
-dataSegmentPusher,
-objectMapper,
-indexIO,
-wrapIndexMerger(indexMerger)
-);
-return appenderator;
+synchronized (this) {
Review comment:
I decided to leave it as is, since the shared portion is pretty small here
and I feel it's more readable without the additional indirection
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website] branch fix-0.15.1-downloads updated (537e271 -> ee38615)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a change to branch fix-0.15.1-downloads
in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git.
from 537e271 fix 0.15.1 download links
add ee38615 oops, add docs
No new revisions were added by this update.
Summary of changes:
.../About-Experimental-Features.html |0
.../Aggregations.html |0
.../{latest => 0.15.1-incubating}/ApproxHisto.html |0
.../Batch-ingestion.html |0
.../Booting-a-production-cluster.html |0
.../Broker-Config.html |0
docs/{latest => 0.15.1-incubating}/Broker.html |0
.../Build-from-source.html |0
.../Cassandra-Deep-Storage.html|0
.../Cluster-setup.html |0
docs/{latest => 0.15.1-incubating}/Compute.html|0
.../Concepts-and-Terminology.html |0
.../Configuration.html |0
docs/{latest => 0.15.1-incubating}/Contribute.html |0
.../Coordinator-Config.html|0
.../{latest => 0.15.1-incubating}/Coordinator.html |0
docs/{latest => 0.15.1-incubating}/DataSource.html |0
.../DataSourceMetadataQuery.html |0
.../Data_formats.html |0
.../Deep-Storage.html |0
docs/{latest => 0.15.1-incubating}/Design.html |0
.../DimensionSpecs.html|0
docs/{latest => 0.15.1-incubating}/Download.html |0
.../Druid-Personal-Demo-Cluster.html |0
.../Druid-vs-Cassandra.html|0
.../Druid-vs-Elasticsearch.html|0
.../Druid-vs-Hadoop.html |0
.../Druid-vs-Impala-or-Shark.html |0
.../Druid-vs-Redshift.html |0
.../Druid-vs-Spark.html|0
.../Druid-vs-Vertica.html |0
docs/{latest => 0.15.1-incubating}/Evaluate.html |0
docs/{latest => 0.15.1-incubating}/Examples.html |0
docs/{latest => 0.15.1-incubating}/Filters.html|0
docs/{latest => 0.15.1-incubating}/Firehose.html |0
.../GeographicQueries.html |0
.../Granularities.html |0
.../GroupByQuery.html |0
.../Hadoop-Configuration.html |0
docs/{latest => 0.15.1-incubating}/Having.html |0
.../Historical-Config.html |0
docs/{latest => 0.15.1-incubating}/Historical.html |0
docs/{latest => 0.15.1-incubating}/Home.html |0
.../Including-Extensions.html |0
.../Indexing-Service-Config.html |0
.../Indexing-Service.html |0
.../Ingestion-FAQ.html |0
.../Ingestion-overview.html|0
docs/{latest => 0.15.1-incubating}/Ingestion.html |0
.../Integrating-Druid-With-Other-Technologies.html |0
.../{latest => 0.15.1-incubating}/Kafka-Eight.html |0
docs/{latest => 0.15.1-incubating}/Libraries.html |0
docs/{latest => 0.15.1-incubating}/LimitSpec.html |0
.../Loading-Your-Data.html |0
docs/{latest => 0.15.1-incubating}/Logging.html|0
docs/{latest => 0.15.1-incubating}/Master.html |0
.../Metadata-storage.html |0
docs/{latest => 0.15.1-incubating}/Metrics.html|0
.../Middlemanager.html |0
docs/{latest => 0.15.1-incubating}/Modules.html|0
docs/{latest => 0.15.1-incubating}/MySQL.html |0
docs/{latest => 0.15.1-incubating}/OrderBy.html|0
.../Other-Hadoop.html |0
.../Papers-and-talks.html |0
docs/{latest => 0.15.1-incubating}/Peons.html |0
.../Performance-FAQ.html |0
docs/{latest => 0.15.1-incubating}/Plumber.html|0
.../Post-aggregations.html |0
.../Production-Cluster-Configuration.html |0
.../Query-Context.html |0
.../Querying-your-data.html|0
docs/{latest => 0.15.1-incubating}/Querying.html |0
.../Realtime-Config.html |0
.../Realtime-ingestion.html|0
docs/{latest => 0.15.1-incubating}/Realtime.html |0
.../Recommendations.html |0
.../Rolling-Updates.html |0
docs/{latest => 0.15.1-incubating}/Router.html |0
.../Rule-Configuration.html
[GitHub] [incubator-druid-website] clintropolis opened a new pull request #30: fix 0.15.1 download links
clintropolis opened a new pull request #30: fix 0.15.1 download links URL: https://github.com/apache/incubator-druid-website/pull/30 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jon-wei commented on a change in pull request #8287: Cleanup appenderators and
segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314547893
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -248,6 +289,50 @@ private AppenderatorConfig
rewriteAppenderatorConfigMemoryLimits(AppenderatorCon
return new MemoryParameterOverridingAppenderatorConfig(baseConfig,
perWorkerLimit);
}
+ private class DatasourceBundle
+ {
+private final SinkQuerySegmentWalker walker;
+private final Map> taskAppenderatorMap;
Review comment:
The CompactionTask creates multiple IndexTask instances and runs them in
sequence (each will create its own appenderator). I updated the CompactionTask
to use the same task ID for the index tasks it runs internally.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314498722
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -248,6 +289,50 @@ private AppenderatorConfig
rewriteAppenderatorConfigMemoryLimits(AppenderatorCon
return new MemoryParameterOverridingAppenderatorConfig(baseConfig,
perWorkerLimit);
}
+ private class DatasourceBundle
+ {
+private final SinkQuerySegmentWalker walker;
+private final Map> taskAppenderatorMap;
+
+public DatasourceBundle(
+String dataSource
+)
+{
+ this.taskAppenderatorMap = new HashMap<>();
+
+ VersionedIntervalTimeline sinkTimeline = new
VersionedIntervalTimeline<>(
+ String.CASE_INSENSITIVE_ORDER
Review comment:
Hmm, I'm not sure what is the proper comparator for the timeline. Raised
https://github.com/apache/incubator-druid/issues/8320.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314501737
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -183,22 +192,48 @@ public Appenderator createOfflineAppenderatorForTask(
IndexMerger indexMerger
)
{
-Appenderator appenderator = Appenderators.createOffline(
-schema,
-rewriteAppenderatorConfigMemoryLimits(config),
-metrics,
-dataSegmentPusher,
-objectMapper,
-indexIO,
-wrapIndexMerger(indexMerger)
-);
-return appenderator;
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.computeIfAbsent(
+ schema.getDataSource(),
+ (datasource) -> {
+return new DatasourceBundle(datasource);
+ }
+ );
+
+ Appenderator appenderator = Appenderators.createOffline(
+ schema,
+ rewriteAppenderatorConfigMemoryLimits(config),
+ metrics,
+ dataSegmentPusher,
+ objectMapper,
+ indexIO,
+ wrapIndexMerger(indexMerger)
+ );
+ datasourceBundle.addAppenderator(taskId, appenderator);
+ return appenderator;
+}
}
@Override
- public void removeAppenderatorForTask(String taskId)
+ public void removeAppenderatorsForTask(
+ String taskId,
+ String dataSource
+ )
{
-// nothing to remove presently
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.get(dataSource);
+ if (datasourceBundle == null) {
+LOG.error("Could not find datasource bundle for [%s], task [%s]",
dataSource, taskId);
+ } else {
+List existingAppenderators =
datasourceBundle.taskAppenderatorMap.remove(taskId);
+if (existingAppenderators == null) {
+ LOG.error("Tried to remove appenderators for task [%s] but none were
found.", taskId);
Review comment:
Same about the log level.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314502791
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -248,6 +289,50 @@ private AppenderatorConfig
rewriteAppenderatorConfigMemoryLimits(AppenderatorCon
return new MemoryParameterOverridingAppenderatorConfig(baseConfig,
perWorkerLimit);
}
+ private class DatasourceBundle
+ {
+private final SinkQuerySegmentWalker walker;
+private final Map> taskAppenderatorMap;
+
+public DatasourceBundle(
+String dataSource
+)
+{
+ this.taskAppenderatorMap = new HashMap<>();
+
+ VersionedIntervalTimeline sinkTimeline = new
VersionedIntervalTimeline<>(
+ String.CASE_INSENSITIVE_ORDER
+ );
+ this.walker = new SinkQuerySegmentWalker(
+ dataSource,
+ sinkTimeline,
+ objectMapper,
+ serviceEmitter,
+ queryRunnerFactoryConglomerateProvider.get(),
+ queryExecutorService,
+ Preconditions.checkNotNull(cache, "cache"),
+ cacheConfig,
+ cachePopulatorStats
+ );
+}
+
+public SinkQuerySegmentWalker getWalker()
+{
+ return walker;
+}
+
+public void addAppenderator(String taskId, Appenderator appenderator)
+{
+ taskAppenderatorMap.computeIfAbsent(
+ taskId,
+ (myTaskId) -> {
+return new ArrayList<>();
+ }
+ );
+ taskAppenderatorMap.get(taskId).add(appenderator);
Review comment:
I think the below is better to avoid an extra lookup on the map.
```
taskAppenderatorMap.computeIfAbsent(taskId, myTaskId -> new
ArrayList<>()).add(appenderator);
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314500808
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -183,22 +192,48 @@ public Appenderator createOfflineAppenderatorForTask(
IndexMerger indexMerger
)
{
-Appenderator appenderator = Appenderators.createOffline(
-schema,
-rewriteAppenderatorConfigMemoryLimits(config),
-metrics,
-dataSegmentPusher,
-objectMapper,
-indexIO,
-wrapIndexMerger(indexMerger)
-);
-return appenderator;
+synchronized (this) {
Review comment:
nit: it looks like the common code between
`createRealtimeAppenderatorForTask` and `createOfflineAppenderatorForTask` can
be extracted as a method which can handle all necessary synchronization and
updating datasourceBundle.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314495674
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -248,6 +289,50 @@ private AppenderatorConfig
rewriteAppenderatorConfigMemoryLimits(AppenderatorCon
return new MemoryParameterOverridingAppenderatorConfig(baseConfig,
perWorkerLimit);
}
+ private class DatasourceBundle
+ {
+private final SinkQuerySegmentWalker walker;
+private final Map> taskAppenderatorMap;
+
+public DatasourceBundle(
Review comment:
nit: all methods of this class can be `private`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314499905
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -134,41 +154,30 @@ public Appenderator createRealtimeAppenderatorForTask(
CachePopulatorStats cachePopulatorStats
)
{
-SinkQuerySegmentWalker segmentWalker =
datasourceSegmentWalkers.computeIfAbsent(
-schema.getDataSource(),
-(datasource) -> {
- VersionedIntervalTimeline sinkTimeline = new
VersionedIntervalTimeline<>(
- String.CASE_INSENSITIVE_ORDER
- );
- SinkQuerySegmentWalker datasourceSegmentWalker = new
SinkQuerySegmentWalker(
- schema.getDataSource(),
- sinkTimeline,
- objectMapper,
- emitter,
- conglomerate,
- this.queryExecutorService,
- Preconditions.checkNotNull(this.cache, "cache"),
- this.cacheConfig,
- this.cachePopulatorStats
- );
- return datasourceSegmentWalker;
-}
-);
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.computeIfAbsent(
+ schema.getDataSource(),
+ (datasource) -> {
+return new DatasourceBundle(datasource);
Review comment:
nit: can be `DatasourceBundle::new`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314503463
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -248,6 +289,50 @@ private AppenderatorConfig
rewriteAppenderatorConfigMemoryLimits(AppenderatorCon
return new MemoryParameterOverridingAppenderatorConfig(baseConfig,
perWorkerLimit);
}
+ private class DatasourceBundle
+ {
+private final SinkQuerySegmentWalker walker;
+private final Map> taskAppenderatorMap;
Review comment:
I'm not sure if there's any task type which uses two or more appenderators.
Would you point me out what task is doing it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8287: Cleanup appenderators and segment walkers in UnifiedIndexerAppenderatorsManager
jihoonson commented on a change in pull request #8287: Cleanup appenderators
and segment walkers in UnifiedIndexerAppenderatorsManager
URL: https://github.com/apache/incubator-druid/pull/8287#discussion_r314501613
##
File path:
server/src/main/java/org/apache/druid/segment/realtime/appenderator/UnifiedIndexerAppenderatorsManager.java
##
@@ -183,22 +192,48 @@ public Appenderator createOfflineAppenderatorForTask(
IndexMerger indexMerger
)
{
-Appenderator appenderator = Appenderators.createOffline(
-schema,
-rewriteAppenderatorConfigMemoryLimits(config),
-metrics,
-dataSegmentPusher,
-objectMapper,
-indexIO,
-wrapIndexMerger(indexMerger)
-);
-return appenderator;
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.computeIfAbsent(
+ schema.getDataSource(),
+ (datasource) -> {
+return new DatasourceBundle(datasource);
+ }
+ );
+
+ Appenderator appenderator = Appenderators.createOffline(
+ schema,
+ rewriteAppenderatorConfigMemoryLimits(config),
+ metrics,
+ dataSegmentPusher,
+ objectMapper,
+ indexIO,
+ wrapIndexMerger(indexMerger)
+ );
+ datasourceBundle.addAppenderator(taskId, appenderator);
+ return appenderator;
+}
}
@Override
- public void removeAppenderatorForTask(String taskId)
+ public void removeAppenderatorsForTask(
+ String taskId,
+ String dataSource
+ )
{
-// nothing to remove presently
+synchronized (this) {
+ DatasourceBundle datasourceBundle = datasourceBundles.get(dataSource);
+ if (datasourceBundle == null) {
+LOG.error("Could not find datasource bundle for [%s], task [%s]",
dataSource, taskId);
Review comment:
Wondering how fatal this case is. I think, if it's very fatal, it should be
propagated to the task so that it will be logged in the task log file in the
future. Otherwise, perhaps the warning level would be enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] stale[bot] commented on issue #7792: Fix wrong order when ordering by string first/last agg with GroupBy V2
stale[bot] commented on issue #7792: Fix wrong order when ordering by string first/last agg with GroupBy V2 URL: https://github.com/apache/incubator-druid/pull/7792#issuecomment-521809619 This pull request/issue is no longer marked as stale. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Keep track of task location for completed tasks (#8286)
This is an automated email from the ASF dual-hosted git repository.
jonwei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new ef7b960 Keep track of task location for completed tasks (#8286)
ef7b960 is described below
commit ef7b9606f2137a7a724e65c52c28375ce0dff427
Author: Jonathan Wei
AuthorDate: Thu Aug 15 16:57:02 2019 -0500
Keep track of task location for completed tasks (#8286)
* Keep track of task location for completed tasks
* Add TaskLifecycleTest location checks
---
.../java/org/apache/druid/indexer/TaskStatus.java | 41 +++---
.../org/apache/druid/indexer/TaskStatusTest.java | 62 ++
.../druid/indexing/overlord/RemoteTaskRunner.java | 16 ++
.../overlord/SingleTaskBackgroundRunner.java | 6 +++
.../apache/druid/indexing/overlord/TaskQueue.java | 6 ++-
.../apache/druid/indexing/overlord/TaskRunner.java | 6 +++
.../overlord/hrtr/HttpRemoteTaskRunner.java| 11
.../indexing/overlord/http/OverlordResource.java | 4 +-
.../druid/indexing/overlord/TaskLifecycleTest.java | 29 --
9 files changed, 165 insertions(+), 16 deletions(-)
diff --git a/core/src/main/java/org/apache/druid/indexer/TaskStatus.java
b/core/src/main/java/org/apache/druid/indexer/TaskStatus.java
index 6b90fef..a0ffe85 100644
--- a/core/src/main/java/org/apache/druid/indexer/TaskStatus.java
+++ b/core/src/main/java/org/apache/druid/indexer/TaskStatus.java
@@ -25,6 +25,8 @@ import com.fasterxml.jackson.annotation.JsonProperty;
import com.google.common.base.Objects;
import com.google.common.base.Preconditions;
+import javax.annotation.Nullable;
+
/**
* Represents the status of a task from the perspective of the coordinator.
The task may be ongoing
* ({@link #isComplete()} false) or it may be complete ({@link #isComplete()}
true).
@@ -37,32 +39,32 @@ public class TaskStatus
public static TaskStatus running(String taskId)
{
-return new TaskStatus(taskId, TaskState.RUNNING, -1, null);
+return new TaskStatus(taskId, TaskState.RUNNING, -1, null, null);
}
public static TaskStatus success(String taskId)
{
-return new TaskStatus(taskId, TaskState.SUCCESS, -1, null);
+return new TaskStatus(taskId, TaskState.SUCCESS, -1, null, null);
}
public static TaskStatus success(String taskId, String errorMsg)
{
-return new TaskStatus(taskId, TaskState.SUCCESS, -1, errorMsg);
+return new TaskStatus(taskId, TaskState.SUCCESS, -1, errorMsg, null);
}
public static TaskStatus failure(String taskId)
{
-return new TaskStatus(taskId, TaskState.FAILED, -1, null);
+return new TaskStatus(taskId, TaskState.FAILED, -1, null, null);
}
public static TaskStatus failure(String taskId, String errorMsg)
{
-return new TaskStatus(taskId, TaskState.FAILED, -1, errorMsg);
+return new TaskStatus(taskId, TaskState.FAILED, -1, errorMsg, null);
}
public static TaskStatus fromCode(String taskId, TaskState code)
{
-return new TaskStatus(taskId, code, -1, null);
+return new TaskStatus(taskId, code, -1, null, null);
}
// The error message can be large, so truncate it to avoid storing large
objects in zookeeper/metadata storage.
@@ -80,19 +82,22 @@ public class TaskStatus
private final TaskState status;
private final long duration;
private final String errorMsg;
+ private final TaskLocation location;
@JsonCreator
protected TaskStatus(
@JsonProperty("id") String id,
@JsonProperty("status") TaskState status,
@JsonProperty("duration") long duration,
- @JsonProperty("errorMsg") String errorMsg
+ @JsonProperty("errorMsg") String errorMsg,
+ @Nullable @JsonProperty("location") TaskLocation location
)
{
this.id = id;
this.status = status;
this.duration = duration;
this.errorMsg = truncateErrorMsg(errorMsg);
+this.location = location == null ? TaskLocation.unknown() : location;
// Check class invariants.
Preconditions.checkNotNull(id, "id");
@@ -123,6 +128,12 @@ public class TaskStatus
return errorMsg;
}
+ @JsonProperty("location")
+ public TaskLocation getLocation()
+ {
+return location;
+ }
+
/**
* Signals that a task is not yet complete, and is still runnable on a
worker. Exactly one of isRunnable,
* isSuccess, or isFailure will be true at any one time.
@@ -172,7 +183,21 @@ public class TaskStatus
public TaskStatus withDuration(long _duration)
{
-return new TaskStatus(id, status, _duration, errorMsg);
+return new TaskStatus(id, status, _duration, errorMsg, location);
+ }
+
+ public TaskStatus withLocation(TaskLocation location)
+ {
+if (location == null) {
+ location = TaskLocation.unknown();
+}
+return new TaskStatus(
+id,
+status,
+duration,
+
[GitHub] [incubator-druid] jon-wei merged pull request #8286: Keep track of task location for completed tasks
jon-wei merged pull request #8286: Keep track of task location for completed tasks URL: https://github.com/apache/incubator-druid/pull/8286 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7792: Fix wrong order when ordering by string first/last agg with GroupBy V2
jihoonson commented on a change in pull request #7792: Fix wrong order when ordering by string first/last agg with GroupBy V2 URL: https://github.com/apache/incubator-druid/pull/7792#discussion_r314513318 ## File path: processing/src/main/java/org/apache/druid/query/aggregation/AggregatorFactory.java ## @@ -51,6 +52,11 @@ public abstract Comparator getComparator(); + public Comparator makeComparatorWithOrderByColumnSpec(OrderByColumnSpec columnSpec) Review comment: I would suggest to rename this to `getComparatorForFinalOrdering()`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson edited a comment on issue #7363: Implementing dropwizard emitter for druid
jihoonson edited a comment on issue #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#issuecomment-521814221 @nishantmonu51 sorry for the late response. Will take a look sometime this week. Would you please fix the conflicts? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Add a warning about experimental segment locking (#8301)
This is an automated email from the ASF dual-hosted git repository. surekha pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new eeae5d9 Add a warning about experimental segment locking (#8301) eeae5d9 is described below commit eeae5d936501d5a9f5f7d4a555f363f97c002119 Author: Jihoon Son AuthorDate: Thu Aug 15 16:07:59 2019 -0700 Add a warning about experimental segment locking (#8301) * Add a warning about experimental segment locking * fix typo --- docs/content/configuration/index.md| 2 +- docs/content/ingestion/locking-and-priority.md | 8 ++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/content/configuration/index.md b/docs/content/configuration/index.md index 134ff18..4812d66 100644 --- a/docs/content/configuration/index.md +++ b/docs/content/configuration/index.md @@ -907,7 +907,7 @@ These Overlord static configurations can be defined in the `overlord/runtime.pro |`druid.indexer.runner.type`|Choices "local" or "remote". Indicates whether tasks should be run locally or in a distributed environment. Experimental task runner "httpRemote" is also available which is same as "remote" but uses HTTP to interact with Middle Manaters instead of Zookeeper.|local| |`druid.indexer.storage.type`|Choices are "local" or "metadata". Indicates whether incoming tasks should be stored locally (in heap) or in metadata storage. Storing incoming tasks in metadata storage allows for tasks to be resumed if the Overlord should fail.|local| |`druid.indexer.storage.recentlyFinishedThreshold`|A duration of time to store task results.|PT24H| -|`druid.indexer.tasklock.forceTimeChunkLock`|If set, all tasks are enforced to use time chunk lock. If not set, each task automatically chooses a lock type to use. This configuration can be overwritten by setting `forceTimeChunkLock` in the [task context](../ingestion/locking-and-priority.html#task-context). See [Task Locking & Priority](../ingestion/locking-and-priority.html) for more details about locking in tasks.|true| +|`druid.indexer.tasklock.forceTimeChunkLock`|_**Setting this to false is still experimental**_ If set, all tasks are enforced to use time chunk lock. If not set, each task automatically chooses a lock type to use. This configuration can be overwritten by setting `forceTimeChunkLock` in the [task context](../ingestion/locking-and-priority.html#task-context). See [Task Locking & Priority](../ingestion/locking-and-priority.html) for more details about locking in tasks.|true| |`druid.indexer.queue.maxSize`|Maximum number of active tasks at one time.|Integer.MAX_VALUE| |`druid.indexer.queue.startDelay`|Sleep this long before starting Overlord queue management. This can be useful to give a cluster time to re-orient itself after e.g. a widespread network issue.|PT1M| |`druid.indexer.queue.restartDelay`|Sleep this long when Overlord queue management throws an exception before trying again.|PT30S| diff --git a/docs/content/ingestion/locking-and-priority.md b/docs/content/ingestion/locking-and-priority.md index f0ba3e1..eaeab99 100644 --- a/docs/content/ingestion/locking-and-priority.md +++ b/docs/content/ingestion/locking-and-priority.md @@ -40,7 +40,7 @@ are used to determine the overshadow relation between segments as seen below. A segment `s1` overshadows another `s2` if -- `s1` has a higher major version than `s2`. +- `s1` has a higher major version than `s2`, or - `s1` has the same major version and a higher minor version than `s2`. Here are some examples. @@ -71,6 +71,10 @@ For example, a Kafka indexing task and a compaction task can always write segmen The reason for this is because a Kafka indexing task always appends new segments, while a compaction task always overwrites existing segments. The segments created with the segment locking have the _same_ major version and a _higher_ minor version. + +The segment locking is still experimental. It could have unknown bugs which potentially lead to incorrect query results. + + To enable segment locking, you may need to set `forceTimeChunkLock` to `false` in the [task context](#task-context). Once `forceTimeChunkLock` is unset, the task will choose a proper lock type to use automatically. Please note that segment lock is not always available. The most common use case where time chunk lock is enforced is @@ -126,7 +130,7 @@ The task context is used for various individual task configuration. The followin |property|default|description| ||---|---| |taskLockTimeout|30|task lock timeout in millisecond. For more details, see [Locking](#locking).| -|forceTimeChunkLock|true|Force to always use time chunk lock. If not set, each task automatically chooses a lock type to use. If this set, it will overwrite the `druid.indexer.tasklock.forceTimeChunkLock`
[GitHub] [incubator-druid] surekhasaharan merged pull request #8301: Add a warning about experimental segment locking
surekhasaharan merged pull request #8301: Add a warning about experimental segment locking URL: https://github.com/apache/incubator-druid/pull/8301 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid-website-src] clintropolis opened a new pull request #37: update website for 0.15.1-incubating
clintropolis opened a new pull request #37: update website for 0.15.1-incubating URL: https://github.com/apache/incubator-druid-website-src/pull/37 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website] branch 0.15.1-incubating created (now 18e5b3f)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch 0.15.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git. at 18e5b3f update website for 0.15.1-incubating This branch includes the following new commits: new 18e5b3f update website for 0.15.1-incubating The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website-src] branch 0.15.1-incubating created (now 2faf8cd)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch 0.15.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git. at 2faf8cd update website for 0.15.1-incubating This branch includes the following new commits: new 2faf8cd update website for 0.15.1-incubating The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid-website] branch 0.15.1-incubating updated (18e5b3f -> 9f91b15)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch 0.15.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git. discard 18e5b3f update website for 0.15.1-incubating add 9f91b15 0.15.1 incubating This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (18e5b3f) \ N -- N -- N refs/heads/0.15.1-incubating (9f91b15) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: blog/2011/04/30/introducing-druid.html | 2 +- blog/2011/05/20/druid-part-deux.html | 2 +- blog/2012/01/19/scaling-the-druid-data-store.html | 2 +- ...-right-cardinality-estimation-for-big-data.html | 2 +- blog/2012/09/21/druid-bitmap-compression.html | 2 +- ...ond-hadoop-fast-ad-hoc-queries-on-big-data.html | 2 +- blog/2012/10/24/introducing-druid.html | 2 +- .../interactive-queries-meet-real-time-data.html | 2 +- blog/2013/04/03/15-minutes-to-live-druid.html | 2 +- blog/2013/04/03/druid-r-meetup.html| 2 +- blog/2013/04/26/meet-the-druid.html| 2 +- blog/2013/05/10/real-time-for-real.html| 2 +- blog/2013/08/06/twitter-tutorial.html | 2 +- blog/2013/08/30/loading-data.html | 2 +- .../12/the-art-of-approximating-distributions.html | 2 +- blog/2013/09/16/upcoming-events.html | 2 +- .../09/19/launching-druid-with-apache-whirr.html | 2 +- blog/2013/09/20/druid-at-xldb.html | 2 +- blog/2013/11/04/querying-your-data.html| 2 +- blog/2014/02/03/rdruid-and-twitterstream.html | 2 +- ...oglog-optimizations-for-real-world-systems.html | 2 +- blog/2014/03/12/batch-ingestion.html | 2 +- blog/2014/03/17/benchmarking-druid.html| 2 +- blog/2014/04/15/intro-to-pydruid.html | 2 +- ...ff-on-the-rise-of-the-real-time-data-stack.html | 2 +- .../07/23/five-tips-for-a-f-ing-great-logo.html| 2 +- blog/2015/02/20/towards-a-community-led-druid.html | 2 +- blog/2015/11/03/seeking-new-committers.html| 2 +- blog/2016/01/06/announcing-new-committers.html | 2 +- blog/2016/06/28/druid-0-9-1.html | 2 +- blog/2016/12/01/druid-0-9-2.html | 2 +- blog/2017/04/18/druid-0-10-0.html | 2 +- blog/2017/08/22/druid-0-10-1.html | 2 +- blog/2017/12/04/druid-0-11-0.html | 2 +- blog/2018/03/08/druid-0-12-0.html | 2 +- blog/2018/06/08/druid-0-12-1.html | 2 +- blog/index.html| 2 +- community/cla.html | 2 +- community/index.html | 2 +- .../comparisons/druid-vs-elasticsearch.html| 4 +- .../comparisons/druid-vs-key-value.html| 4 +- .../comparisons/druid-vs-kudu.html | 4 +- .../comparisons/druid-vs-redshift.html | 4 +- .../comparisons/druid-vs-spark.html| 4 +- .../comparisons/druid-vs-sql-on-hadoop.html| 4 +- docs/0.13.0-incubating/configuration/index.html| 4 +- docs/0.13.0-incubating/configuration/logging.html | 4 +- docs/0.13.0-incubating/configuration/realtime.html | 4 +- .../dependencies/cassandra-deep-storage.html | 4 +- .../dependencies/deep-storage.html | 4 +- .../dependencies/metadata-storage.html | 4 +- docs/0.13.0-incubating/dependencies/zookeeper.html | 4 +- docs/0.13.0-incubating/design/auth.html| 4 +- docs/0.13.0-incubating/design/broker.html | 4 +- docs/0.13.0-incubating/design/coordinator.html | 4 +- docs/0.13.0-incubating/design/historical.html | 4 +- docs/0.13.0-incubating/design/index.html | 4 +- .../0.13.0-incubating/design/indexing-service.html | 4 +- docs/0.13.0-incubating/design/middlemanager.html | 4 +- docs/0.13.0-incubating/design/overlord.html| 4 +- docs/0.13.0-incubating/design/peons.html | 4 +- docs/0.13.0-incubating/design/plumber.html | 4 +- docs/0.13.0-incubating/design/realtime.html| 4 +- docs/0.13.0-incubating/design/segments.html| 4 +-
[incubator-druid-website-src] branch 0.15.1-incubating updated (ad2ed1c -> 68edf87)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch 0.15.1-incubating in repository https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git. discard ad2ed1c fix it add 68edf87 0.15.1-incubating This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (ad2ed1c) \ N -- N -- N refs/heads/0.15.1-incubating (68edf87) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314530271
##
File path: docs/content/development/extensions-contrib/dropwizard.md
##
@@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to
[include](../../operations/including-extensions.html) `dropwizard-emitter`
extension.
+
+## Introduction
+
+This extension integrates
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics
library with druid so that dropwizard users can easily absorb druid into their
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and
histogram.
+These metrics can be emitted using either Console or JMX reporter.
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the
dropwizard metric type, and desired dimensions for every Druid
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer,
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
+be provided as a JSON file. Additionally, this mapping specifies which
dimensions should be included for each metric.
+If the user does not specify their own JSON file, a [default
mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json)
is used.
+All metrics are expected to be mapped. Metrics which are not mapped will be
ignored.
+Dropwizard metric path is organized using the following schema:
+
+` : { "dimensions" : , "type" : , "timeUnit" : }`
+
+e.g.
+
+`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer",
"timeUnit": "MILLISECONDS"}`
+`"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"}`
+
+
+For most use-cases, the default mapping is sufficient.
+
+### Supported Dropwizard reporters
+
+ JMX Reporter
+Used to report druid metrics via JMX.
+```json
+
+druid.emitter.dropwizard.reporter={"type":"jmx"}
+
+```
+
+ Console Reporter
+Used to print Druid Metrics to console logs.
+
+```json
Review comment:
Same for JSON.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314529718
##
File path: docs/content/development/extensions-contrib/dropwizard.md
##
@@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to
[include](../../operations/including-extensions.html) `dropwizard-emitter`
extension.
+
+## Introduction
+
+This extension integrates
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics
library with druid so that dropwizard users can easily absorb druid into their
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and
histogram.
+These metrics can be emitted using either Console or JMX reporter.
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the
dropwizard metric type, and desired dimensions for every Druid
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer,
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
+be provided as a JSON file. Additionally, this mapping specifies which
dimensions should be included for each metric.
+If the user does not specify their own JSON file, a [default
mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json)
is used.
+All metrics are expected to be mapped. Metrics which are not mapped will be
ignored.
+Dropwizard metric path is organized using the following schema:
+
+` : { "dimensions" : , "type" : , "timeUnit" : }`
+
+e.g.
+
+`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer",
"timeUnit": "MILLISECONDS"}`
Review comment:
nit: you can turn on the JSON syntax by the ` ```json ` block.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314540588
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterModule.java
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.databind.Module;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.inject.Binder;
+import com.google.inject.Injector;
+import com.google.inject.Key;
+import com.google.inject.Provides;
+import com.google.inject.name.Named;
+import com.google.inject.name.Names;
+import org.apache.druid.guice.JsonConfigProvider;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.initialization.DruidModule;
+import org.apache.druid.java.util.emitter.core.Emitter;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class DropwizardEmitterModule implements DruidModule
+{
+ private static final String EMITTER_TYPE = "dropwizard";
+
+ @Override
+ public List getJacksonModules()
+ {
+return Collections.EMPTY_LIST;
+ }
+
+ @Override
+ public void configure(Binder binder)
+ {
+JsonConfigProvider.bind(binder, "druid.emitter." + EMITTER_TYPE,
DropwizardEmitterConfig.class);
+ }
+
+ @Provides
+ @ManageLifecycle
Review comment:
I'm not sure this is necessary. All `Emitter`s are supposed to wrapped by
`ServiceEmitter` to use which is managed by lifecycle.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314532539
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardConverter.java
##
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Strings;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.logger.Logger;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Map;
+
+/**
+ *
+ */
+public class DropwizardConverter
+{
+ private static final Logger log = new Logger(DropwizardConverter.class);
+ private Map metricMap;
+
+ public DropwizardConverter(ObjectMapper mapper, String dimensionMapPath)
+ {
+metricMap = readMap(mapper, dimensionMapPath);
+ }
+
+ @Nullable
+ public DropwizardMetricSpec addFilteredUserDims(
+ String service,
+ String metric,
+ Map userDims,
+ Map builder
+ )
+ {
+ /*
+Find the metric in the map. If we cant find it try to look it up
prefixed by the service name.
+This is because some metrics are reported differently, but with the
same name, from different services.
+ */
+DropwizardMetricSpec metricSpec = null;
+if (metricMap.containsKey(metric)) {
+ metricSpec = metricMap.get(metric);
+} else if (metricMap.containsKey(service + "-" + metric)) {
+ metricSpec = metricMap.get(service + "-" + metric);
+}
+if (metricSpec != null) {
+ for (String dim : metricSpec.getDimensions()) {
+if (userDims.containsKey(dim)) {
+ builder.put(dim, userDims.get(dim).toString());
+}
+ }
+ return metricSpec;
+} else {
+ return null;
+}
+ }
+
+ private Map readMap(ObjectMapper mapper,
String dimensionMapPath)
+ {
+try {
+ InputStream is;
+ if (Strings.isNullOrEmpty(dimensionMapPath)) {
+log.info("Using default metric dimension and types");
+is =
this.getClass().getClassLoader().getResourceAsStream("defaultMetricDimensions.json");
+ } else {
+log.info("Using metric dimensions at types at [%s]", dimensionMapPath);
+is = new FileInputStream(new File(dimensionMapPath));
+ }
+ return mapper.readerFor(new TypeReference>()
+ {
+ }).readValue(is);
+}
+catch (IOException e) {
Review comment:
Hmm, why is `is` not closed on normal exit?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314536471 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| Review comment: Would you please add a link to the "Supported Dropwizard reporters" section here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314530722
##
File path: extensions-contrib/dropwizard-emitter/pom.xml
##
@@ -0,0 +1,79 @@
+
+
+
+http://www.w3.org/2001/XMLSchema-instance;
xmlns="http://maven.apache.org/POM/4.0.0;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+ 4.0.0
+
+
+org.apache.druid
+druid
+0.16.0-incubating-SNAPSHOT
+../../pom.xml
+
+
+ org.apache.druid.extensions.contrib
+ dropwizard-emitter
+ dropwizard-emitter
+ Druid emitter extension to convert druid metric to Dropwizard
metrics
+
+
+
+ org.apache.druid
+ druid-core
+ ${project.parent.version}
+ provided
+
+
+ org.apache.druid
+ druid-server
+ ${project.parent.version}
+ provided
+
+
+ io.dropwizard.metrics
+ metrics-core
+ 3.1.0
Review comment:
Is there any reason to use this particular version? Or can we use the 4.0.0
version which is already in `druid-core` dependency?
If it should be 3.1.0, please add it to `licenses.yaml` file.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314540283
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/reporters/DropwizardJMXReporter.java
##
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard.reporters;
+
+import com.codahale.metrics.JmxReporter;
+import com.codahale.metrics.MetricRegistry;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.druid.emitter.dropwizard.DropwizardReporter;
+
+import java.util.Objects;
+
+@JsonTypeName("jmx")
Review comment:
This can be removed since it's already registered in `DropwizardReporter`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314526914 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,86 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| Review comment: Would you please make the doc more clear that it includes both host and port? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314528384 ## File path: docs/content/development/extensions-contrib/dropwizard.md ## @@ -0,0 +1,85 @@ +--- +layout: doc_page +title: "Dropwizard metrics emitter" +--- + + + +# Dropwizard Emitter + +To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension. + +## Introduction + +This extension integrates [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem. +It accumulates druid metrics as dropwizard metrics, and emits them to various sinks via dropwizard supported reporters. +Currently supported dropwizard metrics types counter, gauge, meter, timer and histogram. +These metrics can be emitted using either Console or JMX reporter. + +## Configuration + +All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`. + +|property|description|required?|default| +||---|-|---| +|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be used.|yes|none| +|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics name|no|none| +|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part of the metric name.|no|yes| +|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the dropwizard metric type, and desired dimensions for every Druid metric|no|Default mapping provided. See below.| +|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)| +|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge metrics to be cached at any time. |no| 100K| + + +### Druid to Dropwizard Event Conversion + +Each metric emitted using Dropwizard must specify a type, one of `[timer, counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to Review comment: is something missing after `histogram, `? Or please tidy it up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314530207
##
File path: docs/content/development/extensions-contrib/dropwizard.md
##
@@ -0,0 +1,85 @@
+---
+layout: doc_page
+title: "Dropwizard metrics emitter"
+---
+
+
+
+# Dropwizard Emitter
+
+To use this extension, make sure to
[include](../../operations/including-extensions.html) `dropwizard-emitter`
extension.
+
+## Introduction
+
+This extension integrates
[Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics
library with druid so that dropwizard users can easily absorb druid into their
monitoring ecosystem.
+It accumulates druid metrics as dropwizard metrics, and emits them to various
sinks via dropwizard supported reporters.
+Currently supported dropwizard metrics types counter, gauge, meter, timer and
histogram.
+These metrics can be emitted using either Console or JMX reporter.
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under
`druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+||---|-|---|
+|`druid.emitter.dropwizard.reporters`|List of dropwizard reporters to be
used.|yes|none|
+|`druid.emitter.dropwizard.prefix`|Optional prefix to be used for metrics
name|no|none|
+|`druid.emitter.dropwizard.includeHost`|Flag to include the hostname as part
of the metric name.|no|yes|
+|`druid.emitter.dropwizard.dimensionMapPath`|Path to JSON file defining the
dropwizard metric type, and desired dimensions for every Druid
metric|no|Default mapping provided. See below.|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will
be forwarded to. |no| empty list (no forwarding)|
+|`druid.emitter.dropwizard.maxGaugeCount`| Maximum size of distinct gauge
metrics to be cached at any time. |no| 100K|
+
+
+### Druid to Dropwizard Event Conversion
+
+Each metric emitted using Dropwizard must specify a type, one of `[timer,
counter, guage, meter, histogram, ]`. Dropwizard Emitter expects this mapping to
+be provided as a JSON file. Additionally, this mapping specifies which
dimensions should be included for each metric.
+If the user does not specify their own JSON file, a [default
mapping](https://github.com/apache/incubator-druid/tree/master/extensions-contrib/dropwizard/src/main/resources/defaultMetricDimensions.json)
is used.
+All metrics are expected to be mapped. Metrics which are not mapped will be
ignored.
+Dropwizard metric path is organized using the following schema:
+
+` : { "dimensions" : , "type" : , "timeUnit" : }`
+
+e.g.
+
+`"query/time" : { "dimensions" : ["dataSource", "type"], "type" : "timer",
"timeUnit": "MILLISECONDS"}`
+`"segment/scan/pending" : { "dimensions" : [], "type" : "gauge"}`
+
+
+For most use-cases, the default mapping is sufficient.
+
+### Supported Dropwizard reporters
+
+ JMX Reporter
+Used to report druid metrics via JMX.
+```json
Review comment:
Not sure what is the correct format here. But I don't think it's JSON.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #7363: Implementing dropwizard emitter for druid
jihoonson commented on a change in pull request #7363: Implementing dropwizard
emitter for druid
URL: https://github.com/apache/incubator-druid/pull/7363#discussion_r314536685
##
File path:
extensions-contrib/dropwizard-emitter/src/main/java/org/apache/druid/emitter/dropwizard/DropwizardEmitterConfig.java
##
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Objects;
+
+
+public class DropwizardEmitterConfig
+{
+ private static int DEFAULT_MAX_GAUGE_COUNT = 100_000;
+ @JsonProperty
+ private final List reporters;
+ @JsonProperty
+ private final String prefix;
+ @JsonProperty
+ private final Boolean includeHost;
+ @JsonProperty
+ private final String dimensionMapPath;
+ @JsonProperty
+ private final List alertEmitters;
+ @JsonProperty
+ private final int maxGaugeCount;
+
+ @JsonCreator
+ public DropwizardEmitterConfig(
+ @JsonProperty("reporters") List reporters,
+ @JsonProperty("prefix") String prefix,
+ @JsonProperty("includeHost") Boolean includeHost,
+ @JsonProperty("dimensionMapPath") String dimensionMapPath,
+ @JsonProperty("alertEmitters") List alertEmitters,
+ @JsonProperty("maxGaugeCount") Integer maxGaugeCount
+ )
+ {
+this.reporters = reporters;
+this.prefix = prefix;
+this.alertEmitters = alertEmitters == null ? Collections.emptyList() :
alertEmitters;
+this.includeHost = includeHost != null ? includeHost : true;
+this.dimensionMapPath = dimensionMapPath;
+this.maxGaugeCount = maxGaugeCount == null ? DEFAULT_MAX_GAUGE_COUNT :
maxGaugeCount;
+ }
+
+ @JsonProperty
+ public List getReporters()
+ {
+return reporters;
+ }
+
+ @JsonProperty
+ public String getPrefix()
+ {
+return prefix;
+ }
+
+ @JsonProperty
+ public Boolean getIncludeHost()
+ {
+return includeHost;
+ }
+
+ @JsonProperty
+ public String getDimensionMapPath()
+ {
+return dimensionMapPath;
+ }
+
+ @JsonProperty
+ public List getAlertEmitters()
+ {
+return alertEmitters;
+ }
+
+ @JsonProperty
+ public int getMaxGaugeCount()
+ {
+return maxGaugeCount;
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+if (this == o) {
+ return true;
+}
+if (o == null || getClass() != o.getClass()) {
+ return false;
+}
+DropwizardEmitterConfig that = (DropwizardEmitterConfig) o;
+return maxGaugeCount == that.maxGaugeCount &&
+ Objects.equals(reporters, that.reporters) &&
+ Objects.equals(prefix, that.prefix) &&
+ Objects.equals(includeHost, that.includeHost) &&
+ Objects.equals(dimensionMapPath, that.dimensionMapPath) &&
+ Objects.equals(alertEmitters, that.alertEmitters);
+ }
+
+ @Override
+ public int hashCode()
+ {
+return Objects.hash(reporters, prefix, includeHost, dimensionMapPath,
alertEmitters, maxGaugeCount);
+ }
+
+ @Override
+ public String toString()
+ {
+return "DropwizardEmitterConfig{" +
+ "reporters=" + reporters +
Review comment:
`toString()` is not implemented for `DropwizardReporter`s.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
