[GitHub] ankit0811 commented on issue #6967: NoClassDefFoundError when using druid-hdfs-storage
ankit0811 commented on issue #6967: NoClassDefFoundError when using druid-hdfs-storage URL: https://github.com/apache/incubator-druid/issues/6967#issuecomment-459941408 The `hadoop-client` is unable to pull in the dependency `hadoop-common` in `extension-core/druid-hdfs-storage` cos of the scope `provided` for `hadoop-common` in the main and `hadoop-indexing` pom One possible solution is to provide the dependency `hadoop-common` with scope as `compile` inside `druid-hdfs-storage` pom Let me know what you think @jon-wei This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] glasser opened a new issue #6989: Native batch ingestion didn't replace existing segment
glasser opened a new issue #6989: Native batch ingestion didn't replace existing segment URL: https://github.com/apache/incubator-druid/issues/6989 We're experimenting with native batch ingestion on our 0.13-incubating cluster for the first time (with a custom firehose reading from files saved to GCS by Secor, with a custom InputRowParser). There was a period of a week where the data source had no data. We ran batch ingestion (index_parallel) over one particular hour (5am-6am on December 16th) and it successfully ingested that hour — a segment showed up in the coordinator, it could be queried, etc. (Our segment granularity is HOUR.) Then we ran it again on the entire 24 hours of December 16th. It ran 24 subtasks (our firehose divides up by hour) and ingested the full day, yay! Except when we look in the coordinator, it now lists 2 segments with identical sizes for the 5am hour that we first tested with. Also both of them are listed with the same version which was from the first batch ingestion, not the version that the other 23 segments have from the second batch ingestion. We did not explicitly specify appendToExisting in our ioConfig but I believe the default is false and looking at the task payload it is expanded to false. Are we doing something wrong if our goal is to replace existing segments? Isn't that what `appendToExisting: false` should do? The bad hour in the coordinator:  The good hour:  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] pzhdfy edited a comment on issue #6690: bugfix: when building materialized-view, if taskCount>1, may cause concurrentModificationException
pzhdfy edited a comment on issue #6690: bugfix: when building
materialized-view, if taskCount>1, may cause concurrentModificationException
URL: https://github.com/apache/incubator-druid/pull/6690#issuecomment-459931967
@jihoonson
Yes, this is guarded by taskLock .
This is just the WRONG USE of hashmap.
if we remove an entry from a hashmap by remove(key), while iterating it,
this will throw a concurrentModificationException, even in a single thread.
ref:https://stackoverflow.com/questions/602636/concurrentmodificationexception-and-a-hashmap
```java
for (Map.Entry entry : runningTasks.entrySet()) {
Optional taskStatus =
taskStorage.getStatus(entry.getValue().getId());
if (!taskStatus.isPresent() || !taskStatus.get().isRunnable()) {
runningTasks.remove(entry.getKey());
runningVersion.remove(entry.getKey());
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] pzhdfy commented on issue #6690: bugfix: when building materialized-view, if taskCount>1, may cause concurrentModificationException
pzhdfy commented on issue #6690: bugfix: when building materialized-view, if
taskCount>1, may cause concurrentModificationException
URL: https://github.com/apache/incubator-druid/pull/6690#issuecomment-459931967
Yes, this is guarded by taskLock .
This is just the WRONG USE of hashmap.
if we remove an entry from a hashmap by remove(key), while iterating it,
this will throw a concurrentModificationException, even in a single thread.
ref:https://stackoverflow.com/questions/602636/concurrentmodificationexception-and-a-hashmap
```java
for (Map.Entry entry : runningTasks.entrySet()) {
Optional taskStatus =
taskStorage.getStatus(entry.getValue().getId());
if (!taskStatus.isPresent() || !taskStatus.get().isRunnable()) {
runningTasks.remove(entry.getKey());
runningVersion.remove(entry.getKey());
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] pzhdfy opened a new pull request #6988: [Improvement] historical fast restart by lazy load columns metadata
pzhdfy opened a new pull request #6988: [Improvement] historical fast restart by lazy load columns metadata URL: https://github.com/apache/incubator-druid/pull/6988 We have large data in druid, historical (12 * 2T SATA HDD) will have over 10k segments and 10TB size. When we want to restart a historical to change configuration or after OOM， it will take 40 minutes , that is too slow. We profile the restart progress, and make a flame graph.  We can see io.druid.segment.IndexIO$V9IndexLoader.deserializeColumn cost most time. So we can make columns metadata lazy load , until it gets first used. After optimize, historical restart will only spend 2 minutes( 20X faster). And the flame graph after optimize is below, we can see load metadata spend little time.  we add a new config druid.segmentCache.lazyLoadOnStart (default is false), whether to do this optimize. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6987: ParallelIndexSupervisorTask: don't warn about a default value
jihoonson commented on issue #6987: ParallelIndexSupervisorTask: don't warn about a default value URL: https://github.com/apache/incubator-druid/pull/6987#issuecomment-459931226 Thanks. LGTM after CI. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leerho commented on issue #6869: [Proposal] Deprecating "approximate histogram" in favor of new sketches
leerho commented on issue #6869: [Proposal] Deprecating "approximate histogram" in favor of new sketches URL: https://github.com/apache/incubator-druid/issues/6869#issuecomment-459925200 Next up is the Moment Sketch ... :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leerho commented on issue #6869: [Proposal] Deprecating "approximate histogram" in favor of new sketches
leerho commented on issue #6869: [Proposal] Deprecating "approximate histogram" in favor of new sketches URL: https://github.com/apache/incubator-druid/issues/6869#issuecomment-459925039 FYI I just finished an accuracy study of the Approximate Histogram. It is really bad! You can find it [here](https://datasketches.github.io/docs/Quantiles/DruidApproxHistogramStudy.html). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253248599
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.concurrent.LifecycleLock;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final boolean isCacheEnabled;
+ @Nullable
+ private final ConcurrentMap publishedSegments;
+ private final ScheduledExecutorService scheduledExec;
+ private final long pollPeriodinMS;
+ private final LifecycleLock lifecycleLock = new LifecycleLock();
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+Preconditions.checkNotNull(plannerConfig, "plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+this.isCacheEnabled = plannerConfig.isMetadataSegmentCacheEnable();
+this.pollPeriodinMS = plannerConfig.getMetadataSegmentPollPeriod();
+this.publishedSegments = isCacheEnabled ? new ConcurrentHashMap<>(1000) :
null;
+this.scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (!lifecycleLock.canStart()) {
+ throw new ISE("can't start.");
+}
+try {
+ if (isCacheEnabled) {
+try {
+ poll();
+}
+finally {
+ scheduledExec.schedule(new PollTask(), 0, Time
[GitHub] glasser opened a new pull request #6987: ParallelIndexSupervisorTask: don't warn about a default value
glasser opened a new pull request #6987: ParallelIndexSupervisorTask: don't warn about a default value URL: https://github.com/apache/incubator-druid/pull/6987 Native batch indexing doesn't yet support the maxParseExceptions, maxSavedParseExceptions, and logParseExceptions tuning config options, so ParallelIndexSupervisorTask logs if these are set. But the default value for maxParseExceptions is Integer.MAX_VALUE, which means that you'll get the maxParseExceptions flavor of this warning even if you don't configure maxParseExceptions. This PR changes all three warnings to occur if you change the settings from the default; this mostly affects the maxParseExceptions warning. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253245250
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -106,10 +106,15 @@ public void start()
}
try {
if (isCacheEnabled) {
-scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
-lifecycleLock.started();
-log.info("MetadataSegmentView Started.");
+try {
+ poll();
+}
+finally {
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
Review comment:
yeah, changed to use `pollPeriodinMS`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253245195
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -106,10 +106,15 @@ public void start()
}
try {
if (isCacheEnabled) {
-scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
-lifecycleLock.started();
-log.info("MetadataSegmentView Started.");
+try {
+ poll();
+}
+finally {
Review comment:
added catch block
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei commented on a change in pull request #6901: Introduce published segment cache in broker
jon-wei commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253244563
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.concurrent.LifecycleLock;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final boolean isCacheEnabled;
+ @Nullable
+ private final ConcurrentMap publishedSegments;
+ private final ScheduledExecutorService scheduledExec;
+ private final long pollPeriodinMS;
+ private final LifecycleLock lifecycleLock = new LifecycleLock();
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+Preconditions.checkNotNull(plannerConfig, "plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+this.isCacheEnabled = plannerConfig.isMetadataSegmentCacheEnable();
+this.pollPeriodinMS = plannerConfig.getMetadataSegmentPollPeriod();
+this.publishedSegments = isCacheEnabled ? new ConcurrentHashMap<>(1000) :
null;
+this.scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (!lifecycleLock.canStart()) {
+ throw new ISE("can't start.");
+}
+try {
+ if (isCacheEnabled) {
+try {
+ poll();
+}
+finally {
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MI
[GitHub] jon-wei commented on a change in pull request #6901: Introduce published segment cache in broker
jon-wei commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253244563
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,258 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.concurrent.LifecycleLock;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final boolean isCacheEnabled;
+ @Nullable
+ private final ConcurrentMap publishedSegments;
+ private final ScheduledExecutorService scheduledExec;
+ private final long pollPeriodinMS;
+ private final LifecycleLock lifecycleLock = new LifecycleLock();
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+Preconditions.checkNotNull(plannerConfig, "plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+this.isCacheEnabled = plannerConfig.isMetadataSegmentCacheEnable();
+this.pollPeriodinMS = plannerConfig.getMetadataSegmentPollPeriod();
+this.publishedSegments = isCacheEnabled ? new ConcurrentHashMap<>(1000) :
null;
+this.scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (!lifecycleLock.canStart()) {
+ throw new ISE("can't start.");
+}
+try {
+ if (isCacheEnabled) {
+try {
+ poll();
+}
+finally {
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MI
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253242662
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -106,10 +106,15 @@ public void start()
}
try {
if (isCacheEnabled) {
-scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
-lifecycleLock.started();
-log.info("MetadataSegmentView Started.");
+try {
+ poll();
+}
+finally {
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
Review comment:
Please set the proper wait time instead of starting poll() immediately.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] justinborromeo opened a new pull request #6986: Create Scan Benchmark
justinborromeo opened a new pull request #6986: Create Scan Benchmark URL: https://github.com/apache/incubator-druid/pull/6986 This PR creates a Scan query benchmark to act as a baseline for future changes to the Scan query (specifically #6088 ). The queries are fairly simple and if anyone has any suggestions for better queries to do, lmk. The ScanQuery builder was also moved from the ScanQuery class to the Druids class to match the pattern of all the other query builders. Not sure if this matters since this PR doesn't involve any changes to the query code but here's the results of the benchmark:  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253242562
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -106,10 +106,15 @@ public void start()
}
try {
if (isCacheEnabled) {
-scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
-lifecycleLock.started();
-log.info("MetadataSegmentView Started.");
+try {
+ poll();
+}
+finally {
Review comment:
Please add a catch block to emit exceptions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
jon-wei commented on a change in pull request #6951: Add more sketch aggregator
support in Druid SQL
URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253238633
##
File path:
extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/sql/HllSketchSqlAggregatorTest.java
##
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.datasketches.hll.sql;
+
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Druids;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.QueryDataSource;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.DoubleSumAggregatorFactory;
+import org.apache.druid.query.aggregation.FilteredAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchBuildAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeAggregatorFactory;
+import org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule;
+import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
+import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
+import
org.apache.druid.query.aggregation.post.FinalizingFieldAccessPostAggregator;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.expression.TestExprMacroTable;
+import org.apache.druid.query.groupby.GroupByQuery;
+import org.apache.druid.query.spec.MultipleIntervalSegmentSpec;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
+import
org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.server.security.AuthenticationResult;
+import org.apache.druid.sql.SqlLifecycle;
+import org.apache.druid.sql.SqlLifecycleFactory;
+import org.apache.druid.sql.calcite.BaseCalciteQueryTest;
+import org.apache.druid.sql.calcite.filtration.Filtration;
+import org.apache.druid.sql.calcite.planner.DruidOperatorTable;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.schema.DruidSchema;
+import org.apache.druid.sql.calcite.schema.SystemSchema;
+import org.apache.druid.sql.calcite.util.CalciteTestBase;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.sql.calcite.util.QueryLogHook;
+import org.apache.druid.sql.calcite.util.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+public class HllSketchSqlAggregatorTest extends CalciteTestBase
+{
+ private static final String DATA_SOURCE = "foo";
+
+ private static QueryRunnerFactoryConglomerate conglomerate;
+ private static Closer resourceCloser;
+ private static AuthenticationResult authenticationR
[GitHub] jon-wei commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
jon-wei commented on a change in pull request #6951: Add more sketch aggregator
support in Druid SQL
URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253238585
##
File path:
extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/sql/HllSketchSqlAggregatorTest.java
##
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.datasketches.hll.sql;
+
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Druids;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.QueryDataSource;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.DoubleSumAggregatorFactory;
+import org.apache.druid.query.aggregation.FilteredAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchBuildAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeAggregatorFactory;
+import org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule;
+import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
+import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
+import
org.apache.druid.query.aggregation.post.FinalizingFieldAccessPostAggregator;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.expression.TestExprMacroTable;
+import org.apache.druid.query.groupby.GroupByQuery;
+import org.apache.druid.query.spec.MultipleIntervalSegmentSpec;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
+import
org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.server.security.AuthenticationResult;
+import org.apache.druid.sql.SqlLifecycle;
+import org.apache.druid.sql.SqlLifecycleFactory;
+import org.apache.druid.sql.calcite.BaseCalciteQueryTest;
+import org.apache.druid.sql.calcite.filtration.Filtration;
+import org.apache.druid.sql.calcite.planner.DruidOperatorTable;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.schema.DruidSchema;
+import org.apache.druid.sql.calcite.schema.SystemSchema;
+import org.apache.druid.sql.calcite.util.CalciteTestBase;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.sql.calcite.util.QueryLogHook;
+import org.apache.druid.sql.calcite.util.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+public class HllSketchSqlAggregatorTest extends CalciteTestBase
+{
+ private static final String DATA_SOURCE = "foo";
+
+ private static QueryRunnerFactoryConglomerate conglomerate;
+ private static Closer resourceCloser;
+ private static AuthenticationResult authenticationR
[GitHub] michael-trelinski commented on a change in pull request #6740: Zookeeper loss
michael-trelinski commented on a change in pull request #6740: Zookeeper loss
URL: https://github.com/apache/incubator-druid/pull/6740#discussion_r253238507
##
File path: server/src/main/java/org/apache/druid/curator/CuratorConfig.java
##
@@ -109,4 +113,14 @@ public String getAuthScheme()
return authScheme;
}
+ public boolean getTerminateDruidProcessOnConnectFail()
+ {
+return terminateDruidProcessOnConnectFail;
+ }
+
+ public void setTerminateDruidProcessOnConnectFail(Boolean
terminateDruidProcessOnConnectFail)
+ {
+this.terminateDruidProcessOnConnectFail =
terminateDruidProcessOnConnectFail == null ? false :
terminateDruidProcessOnConnectFail;
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] michael-trelinski commented on a change in pull request #6740: Zookeeper loss
michael-trelinski commented on a change in pull request #6740: Zookeeper loss
URL: https://github.com/apache/incubator-druid/pull/6740#discussion_r253238376
##
File path: server/src/main/java/org/apache/druid/curator/CuratorModule.java
##
@@ -127,6 +154,29 @@ public EnsembleProvider
makeEnsembleProvider(CuratorConfig config, ExhibitorConf
return new FixedEnsembleProvider(config.getZkHosts());
}
+RetryPolicy retryPolicy;
+if (config.getTerminateDruidProcess()) {
+ final Function exitFunction = new Function()
+ {
+@Override
+public Void apply(Void aVoid)
+{
+ log.error("Zookeeper can't be reached, forcefully stopping virtual
machine...");
+ System.exit(1);
Review comment:
Switched to this pattern. I didn't think it was necessary since I've never
seen a logger kill a JVM.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] michael-trelinski commented on a change in pull request #6740: Zookeeper loss
michael-trelinski commented on a change in pull request #6740: Zookeeper loss
URL: https://github.com/apache/incubator-druid/pull/6740#discussion_r253238297
##
File path: server/src/main/java/org/apache/druid/curator/CuratorModule.java
##
@@ -79,10 +80,24 @@ public CuratorFramework makeCurator(CuratorConfig config,
EnsembleProvider ensem
StringUtils.format("%s:%s", config.getZkUser(),
config.getZkPwd()).getBytes(StandardCharsets.UTF_8)
);
}
+
+final Function exitFunction = new Function()
+{
+ @Override
+ public Void apply(Void someVoid)
+ {
+log.error("Zookeeper can't be reached, forcefully stopping
lifecycle...");
+lifecycle.stop();
Review comment:
It is clearly logged:
makeCurator()'s exit will look like this:
> Zookeeper can't be reached, forcefully stopping lifecycle...
> Zookeeper can't be reached, forcefully stopping virtual machine...
makeEnsembleProvider()'s exit will look like this:
> Zookeeper can't be reached, forcefully stopping virtual machine...
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6690: bugfix: when building materialized-view, if taskCount>1, may cause concurrentModificationException
jihoonson commented on issue #6690: bugfix: when building materialized-view, if
taskCount>1, may cause concurrentModificationException
URL: https://github.com/apache/incubator-druid/pull/6690#issuecomment-459907344
@pzhdfy thanks for the patch and sorry for late review. Do you know what
threads can read and remove at the same time? It's currently guarded by
`taskLock`.
```java
synchronized (taskLock) {
for (Map.Entry entry :
runningTasks.entrySet()) {
Optional taskStatus =
taskStorage.getStatus(entry.getValue().getId());
if (!taskStatus.isPresent() || !taskStatus.get().isRunnable()) {
runningTasks.remove(entry.getKey());
runningVersion.remove(entry.getKey());
}
}
if (runningTasks.size() == maxTaskCount) {
//if the number of running tasks reach the max task count,
supervisor won't submit new tasks.
return;
}
Pair, Map>>
toBuildIntervalAndBaseSegments = checkSegments();
SortedMap sortedToBuildVersion =
toBuildIntervalAndBaseSegments.lhs;
Map> baseSegments =
toBuildIntervalAndBaseSegments.rhs;
missInterval = sortedToBuildVersion.keySet();
submitTasks(sortedToBuildVersion, baseSegments);
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
jihoonson commented on a change in pull request #6951: Add more sketch
aggregator support in Druid SQL
URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253233621
##
File path:
extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/sql/HllSketchSqlAggregatorTest.java
##
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.datasketches.hll.sql;
+
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Druids;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.QueryDataSource;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.DoubleSumAggregatorFactory;
+import org.apache.druid.query.aggregation.FilteredAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchBuildAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeAggregatorFactory;
+import org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule;
+import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
+import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
+import
org.apache.druid.query.aggregation.post.FinalizingFieldAccessPostAggregator;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.expression.TestExprMacroTable;
+import org.apache.druid.query.groupby.GroupByQuery;
+import org.apache.druid.query.spec.MultipleIntervalSegmentSpec;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
+import
org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.server.security.AuthenticationResult;
+import org.apache.druid.sql.SqlLifecycle;
+import org.apache.druid.sql.SqlLifecycleFactory;
+import org.apache.druid.sql.calcite.BaseCalciteQueryTest;
+import org.apache.druid.sql.calcite.filtration.Filtration;
+import org.apache.druid.sql.calcite.planner.DruidOperatorTable;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.schema.DruidSchema;
+import org.apache.druid.sql.calcite.schema.SystemSchema;
+import org.apache.druid.sql.calcite.util.CalciteTestBase;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.sql.calcite.util.QueryLogHook;
+import org.apache.druid.sql.calcite.util.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+public class HllSketchSqlAggregatorTest extends CalciteTestBase
+{
+ private static final String DATA_SOURCE = "foo";
+
+ private static QueryRunnerFactoryConglomerate conglomerate;
+ private static Closer resourceCloser;
+ private static AuthenticationResult authenticatio
[GitHub] jihoonson commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
jihoonson commented on a change in pull request #6951: Add more sketch
aggregator support in Druid SQL
URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253233599
##
File path:
extensions-core/datasketches/src/test/java/org/apache/druid/query/aggregation/datasketches/hll/sql/HllSketchSqlAggregatorTest.java
##
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.datasketches.hll.sql;
+
+import com.fasterxml.jackson.databind.Module;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import org.apache.druid.common.config.NullHandling;
+import org.apache.druid.java.util.common.Pair;
+import org.apache.druid.java.util.common.granularity.Granularities;
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.query.Druids;
+import org.apache.druid.query.Query;
+import org.apache.druid.query.QueryDataSource;
+import org.apache.druid.query.QueryRunnerFactoryConglomerate;
+import org.apache.druid.query.aggregation.CountAggregatorFactory;
+import org.apache.druid.query.aggregation.DoubleSumAggregatorFactory;
+import org.apache.druid.query.aggregation.FilteredAggregatorFactory;
+import org.apache.druid.query.aggregation.LongSumAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchBuildAggregatorFactory;
+import
org.apache.druid.query.aggregation.datasketches.hll.HllSketchMergeAggregatorFactory;
+import org.apache.druid.query.aggregation.datasketches.hll.HllSketchModule;
+import org.apache.druid.query.aggregation.post.ArithmeticPostAggregator;
+import org.apache.druid.query.aggregation.post.FieldAccessPostAggregator;
+import
org.apache.druid.query.aggregation.post.FinalizingFieldAccessPostAggregator;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.expression.TestExprMacroTable;
+import org.apache.druid.query.groupby.GroupByQuery;
+import org.apache.druid.query.spec.MultipleIntervalSegmentSpec;
+import org.apache.druid.segment.IndexBuilder;
+import org.apache.druid.segment.QueryableIndex;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.incremental.IncrementalIndexSchema;
+import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
+import
org.apache.druid.segment.writeout.OffHeapMemorySegmentWriteOutMediumFactory;
+import org.apache.druid.server.security.AuthTestUtils;
+import org.apache.druid.server.security.AuthenticationResult;
+import org.apache.druid.sql.SqlLifecycle;
+import org.apache.druid.sql.SqlLifecycleFactory;
+import org.apache.druid.sql.calcite.BaseCalciteQueryTest;
+import org.apache.druid.sql.calcite.filtration.Filtration;
+import org.apache.druid.sql.calcite.planner.DruidOperatorTable;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.planner.PlannerFactory;
+import org.apache.druid.sql.calcite.schema.DruidSchema;
+import org.apache.druid.sql.calcite.schema.SystemSchema;
+import org.apache.druid.sql.calcite.util.CalciteTestBase;
+import org.apache.druid.sql.calcite.util.CalciteTests;
+import org.apache.druid.sql.calcite.util.QueryLogHook;
+import org.apache.druid.sql.calcite.util.SpecificSegmentsQuerySegmentWalker;
+import org.apache.druid.timeline.DataSegment;
+import org.apache.druid.timeline.partition.LinearShardSpec;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+public class HllSketchSqlAggregatorTest extends CalciteTestBase
+{
+ private static final String DATA_SOURCE = "foo";
+
+ private static QueryRunnerFactoryConglomerate conglomerate;
+ private static Closer resourceCloser;
+ private static AuthenticationResult authenticatio
[GitHub] ankit0811 commented on issue #6967: NoClassDefFoundError when using druid-hdfs-storage
ankit0811 commented on issue #6967: NoClassDefFoundError when using druid-hdfs-storage URL: https://github.com/apache/incubator-druid/issues/6967#issuecomment-459896068 can take a look to check if its related to #6828 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: lol (#6985)
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 6430ef8 lol (#6985) 6430ef8 is described below commit 6430ef8e1b62515576f5c7ec090bcd79da7eb8ee Author: Justin Borromeo AuthorDate: Fri Feb 1 14:21:13 2019 -0800 lol (#6985) --- docs/content/ingestion/schema-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/ingestion/schema-design.md b/docs/content/ingestion/schema-design.md index 0408a39..58f0fae 100644 --- a/docs/content/ingestion/schema-design.md +++ b/docs/content/ingestion/schema-design.md @@ -86,7 +86,7 @@ rollup and load your existing data as-is. Rollup in Druid is similar to creating (Like OpenTSDB or InfluxDB.) Similar to time series databases, Druid's data model requires a timestamp. Druid is not a timeseries database, but -it is a natural choice for storing timeseries data. Its flexible data mdoel allows it to store both timeseries and +it is a natural choice for storing timeseries data. Its flexible data model allows it to store both timeseries and non-timeseries data, even in the same datasource. To achieve best-case compression and query performance in Druid for timeseries data, it is important to partition and - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] fjy merged pull request #6985: Fix typo in schema design doc
fjy merged pull request #6985: Fix typo in schema design doc URL: https://github.com/apache/incubator-druid/pull/6985 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] justinborromeo opened a new pull request #6985: Fix typo in schema design doc
justinborromeo opened a new pull request #6985: Fix typo in schema design doc URL: https://github.com/apache/incubator-druid/pull/6985 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei opened a new issue #6984: Need to fix LICENSE and NOTICE for next release
jon-wei opened a new issue #6984: Need to fix LICENSE and NOTICE for next release URL: https://github.com/apache/incubator-druid/issues/6984 As brought up during the vote for 0.13.0-incubating, we will need to correct our LICENSE and NOTICE files for the next release. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253214293
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.concurrent.LifecycleLock;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final boolean isCacheEnabled;
+ private final ConcurrentMap publishedSegments;
+ private final ScheduledExecutorService scheduledExec;
+ private final long pollPeriodinMS;
+ private LifecycleLock lifecycleLock = new LifecycleLock();
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+Preconditions.checkNotNull(plannerConfig, "plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+this.isCacheEnabled = plannerConfig.isMetadataSegmentCacheEnable();
+this.pollPeriodinMS = plannerConfig.getMetadataSegmentPollPeriod();
+this.publishedSegments = isCacheEnabled ? new ConcurrentHashMap<>(1000) :
null;
+this.scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (!lifecycleLock.canStart()) {
+ throw new ISE("can't start.");
+}
+try {
+ if (isCacheEnabled) {
+scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+lifecycleLock.started();
+log.info("MetadataSegmentView Started.");
+ }
+}
[incubator-druid] branch master updated: Fix node path for building the unified console (#6981)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 7d4cc28 Fix node path for building the unified console (#6981) 7d4cc28 is described below commit 7d4cc287306e9aa576e29b722a0d74f6a86fdab8 Author: Jihoon Son AuthorDate: Fri Feb 1 13:57:17 2019 -0800 Fix node path for building the unified console (#6981) --- web-console/script/build | 8 ++-- 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/web-console/script/build b/web-console/script/build index 52e35f5..147f0e6 100755 --- a/web-console/script/build +++ b/web-console/script/build @@ -18,19 +18,15 @@ set -e -#pushd web-console - echo "Copying coordinator console in..." cp -r ./node_modules/druid-console/coordinator-console . cp -r ./node_modules/druid-console/pages . cp ./node_modules/druid-console/index.html . echo "Transpiling ReactTable CSS..." -./node_modules/.bin/stylus lib/react-table.styl -o lib/react-table.css +PATH="./target/node:$PATH" ./node_modules/.bin/stylus lib/react-table.styl -o lib/react-table.css echo "Webpacking everything..." -./node_modules/.bin/webpack -c webpack.config.js +PATH="./target/node:$PATH" ./node_modules/.bin/webpack -c webpack.config.js echo "Done! Have a good day." - -#popd - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis merged pull request #6981: Fix node path for building the unified console
clintropolis merged pull request #6981: Fix node path for building the unified console URL: https://github.com/apache/incubator-druid/pull/6981 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei merged pull request #6950: bloom filter sql aggregator
jon-wei merged pull request #6950: bloom filter sql aggregator URL: https://github.com/apache/incubator-druid/pull/6950 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: bloom filter sql aggregator (#6950)
This is an automated email from the ASF dual-hosted git repository.
jonwei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 7a5827e bloom filter sql aggregator (#6950)
7a5827e is described below
commit 7a5827e12eb65eef80e08fe86ef76604019d6af8
Author: Clint Wylie
AuthorDate: Fri Feb 1 13:54:46 2019 -0800
bloom filter sql aggregator (#6950)
* adds sql aggregator for bloom filter, adds complex value serde for sql
results
* fix tests
* checkstyle
* fix copy-paste
---
docs/content/configuration/index.md| 1 +
.../development/extensions-core/bloom-filter.md| 25 +-
docs/content/querying/sql.md | 1 +
.../druid/guice/BloomFilterExtensionModule.java| 3 +-
.../bloom/sql/BloomFilterSqlAggregator.java| 212 +++
.../apache/druid/query/filter/BloomKFilter.java| 2 +-
.../bloom/sql/BloomFilterSqlAggregatorTest.java| 642 +
.../druid/sql/calcite/planner/PlannerConfig.java | 14 +-
.../apache/druid/sql/calcite/rel/QueryMaker.java | 14 +-
.../druid/sql/calcite/BaseCalciteQueryTest.java| 8 +
.../apache/druid/sql/calcite/CalciteQueryTest.java | 17 +-
.../druid/sql/calcite/http/SqlResourceTest.java| 9 +-
12 files changed, 936 insertions(+), 12 deletions(-)
diff --git a/docs/content/configuration/index.md
b/docs/content/configuration/index.md
index 990f9ce..221cccd 100644
--- a/docs/content/configuration/index.md
+++ b/docs/content/configuration/index.md
@@ -1418,6 +1418,7 @@ The Druid SQL server is configured through the following
properties on the Broke
|`druid.sql.planner.useFallback`|Whether to evaluate operations on the Broker
when they cannot be expressed as Druid queries. This option is not recommended
for production since it can generate unscalable query plans. If false, SQL
queries that cannot be translated to Druid queries will fail.|false|
|`druid.sql.planner.requireTimeCondition`|Whether to require SQL to have
filter conditions on __time column so that all generated native queries will
have user specified intervals. If true, all queries wihout filter condition on
__time column will fail|false|
|`druid.sql.planner.sqlTimeZone`|Sets the default time zone for the server,
which will affect how time functions and timestamp literals behave. Should be a
time zone name like "America/Los_Angeles" or offset like "-08:00".|UTC|
+|`druid.sql.planner.serializeComplexValues`|Whether to serialize "complex"
output values, false will return the class name instead of the serialized
value.|true|
Broker Caching
diff --git a/docs/content/development/extensions-core/bloom-filter.md
b/docs/content/development/extensions-core/bloom-filter.md
index f878e75..651cc30 100644
--- a/docs/content/development/extensions-core/bloom-filter.md
+++ b/docs/content/development/extensions-core/bloom-filter.md
@@ -89,9 +89,9 @@ This string can then be used in the native or sql Druid query.
Note: `org.apache.hive.common.util.BloomKFilter` provides a serialize method
which can be used to serialize bloom filters to outputStream.
-### SQL Queries
+### Filtering SQL Queries
-Bloom filters are supported in SQL via the `bloom_filter_test` operator:
+Bloom filters can be used in SQL `WHERE` clauses via the `bloom_filter_test`
operator:
```sql
SELECT COUNT(*) FROM druid.foo WHERE bloom_filter_test(,
'')
@@ -108,7 +108,11 @@ bloom_filter_test(,
'')
## Bloom Filter Query Aggregator
-Input for a `bloomKFilter` can also be created from a druid query with the
`bloom` aggregator.
+Input for a `bloomKFilter` can also be created from a druid query with the
`bloom` aggregator. Note that it is very
+important to set a reasonable value for the `maxNumEntries` parameter, which
is the maximum number of distinct entries
+that the bloom filter can represent without increasing the false postive rate.
It may be worth performing a query using
+one of the unique count sketches to calculate the value for this parameter in
order to build a bloom filter appropriate
+for the query.
### JSON Specification of Bloom Filter Aggregator
@@ -157,8 +161,19 @@ response
[{"timestamp":"2015-09-12T00:00:00.000Z","result":{"userBloom":"BAAAJh..."}}]
```
-These values can then be set in the filter specification above.
+These values can then be set in the filter specification described above.
Ordering results by a bloom filter aggregator, for example in a TopN query,
will perform a comparatively expensive
linear scan _of the filter itself_ to count the number of set bits as a means
of approximating how many items have been
-added to the set. As such, ordering by an alternate aggregation is recommended
if possible.
\ No newline at end of file
+added to the set. As such, ordering by an alternate aggregation is recommended
if
[GitHub] jorbay-au commented on issue #6983: Remove repeated word in indexing-service.md
jorbay-au commented on issue #6983: Remove repeated word in indexing-service.md URL: https://github.com/apache/incubator-druid/pull/6983#issuecomment-459877682 No problem! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei merged pull request #6976: Updated metrics.md to fix format
jon-wei merged pull request #6976: Updated metrics.md to fix format URL: https://github.com/apache/incubator-druid/pull/6976 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Update metrics.md (#6976)
This is an automated email from the ASF dual-hosted git repository. jonwei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new e45f9ea Update metrics.md (#6976) e45f9ea is described below commit e45f9ea5e9d28e2fd08799696b4722eb525d5376 Author: lxqfy AuthorDate: Sat Feb 2 05:40:44 2019 +0800 Update metrics.md (#6976) --- docs/content/operations/metrics.md | 7 ++- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/content/operations/metrics.md b/docs/content/operations/metrics.md index 7bce8c0..e8e8958 100644 --- a/docs/content/operations/metrics.md +++ b/docs/content/operations/metrics.md @@ -154,8 +154,7 @@ These metrics are only available if the RealtimeMetricsMonitor is included in th |`ingest/events/thrownAway`|Number of events rejected because they are outside the windowPeriod.|dataSource, taskId, taskType.|0| |`ingest/events/unparseable`|Number of events rejected because the events are unparseable.|dataSource, taskId, taskType.|0| |`ingest/events/duplicate`|Number of events rejected because the events are duplicated.|dataSource, taskId, taskType.|0| -|`ingest/events/processed`|Number of events successfully processed per emission period.|dataSource, taskId, taskType.|Equal to your # of events per -emission period.| +|`ingest/events/processed`|Number of events successfully processed per emission period.|dataSource, taskId, taskType.|Equal to your # of events per emission period.| |`ingest/rows/output`|Number of Druid rows persisted.|dataSource, taskId, taskType.|Your # of events with rollup.| |`ingest/persists/count`|Number of times persist occurred.|dataSource, taskId, taskType.|Depends on configuration.| |`ingest/persists/time`|Milliseconds spent doing intermediate persist.|dataSource, taskId, taskType.|Depends on configuration. Generally a few minutes at most.| @@ -252,9 +251,7 @@ The following metric is only available if the EventReceiverFirehoseMonitor modul |Metric|Description|Dimensions|Normal Value| |--|---|--|| -|`ingest/events/buffered`|Number of events queued in the EventReceiverFirehose's buffer|serviceName, dataSource, taskId, taskType, bufferCapacity -.|Equal -to current # of events in the buffer queue.| +|`ingest/events/buffered`|Number of events queued in the EventReceiverFirehose's buffer|serviceName, dataSource, taskId, taskType, bufferCapacity.|Equal to current # of events in the buffer queue.| |`ingest/bytes/received`|Number of bytes received by the EventReceiverFirehose.|serviceName, dataSource, taskId, taskType.|Varies.| ## Sys - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Remove repeated word in indexing-service.md (#6983)
This is an automated email from the ASF dual-hosted git repository. jonwei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git The following commit(s) were added to refs/heads/master by this push: new 852fe86 Remove repeated word in indexing-service.md (#6983) 852fe86 is described below commit 852fe86ea2b361a97d1fe0770bc2457a4809d86e Author: jorbay-au <40071830+jorbay...@users.noreply.github.com> AuthorDate: Fri Feb 1 13:38:22 2019 -0800 Remove repeated word in indexing-service.md (#6983) --- docs/content/design/indexing-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/design/indexing-service.md b/docs/content/design/indexing-service.md index 0d583b7..8a3d5a6 100644 --- a/docs/content/design/indexing-service.md +++ b/docs/content/design/indexing-service.md @@ -26,7 +26,7 @@ title: "Indexing Service" The indexing service is a highly-available, distributed service that runs indexing related tasks. -Indexing tasks [tasks](../ingestion/tasks.html) create (and sometimes destroy) Druid [segments](../design/segments.html). The indexing service has a master/slave like architecture. +Indexing [tasks](../ingestion/tasks.html) create (and sometimes destroy) Druid [segments](../design/segments.html). The indexing service has a master/slave like architecture. The indexing service is composed of three main components: a [Peon](../design/peons.html) component that can run a single task, a [Middle Manager](../design/middlemanager.html) component that manages Peons, and an [Overlord](../design/overlord.html) component that manages task distribution to MiddleManagers. Overlords and MiddleManagers may run on the same node or across multiple nodes while MiddleManagers and Peons always run on the same node. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei merged pull request #6983: Remove repeated word in indexing-service.md
jon-wei merged pull request #6983: Remove repeated word in indexing-service.md URL: https://github.com/apache/incubator-druid/pull/6983 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jorbay-au opened a new pull request #6983: Remove repeated word in indexing-service.md
jorbay-au opened a new pull request #6983: Remove repeated word in indexing-service.md URL: https://github.com/apache/incubator-druid/pull/6983 `tasks` was written twice. I spotted this while taking a dive into the documentation so deep I've read the preambles. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei opened a new issue #6982: PreResponseAuthorizationCheckFilter error when loading coordinator console
jon-wei opened a new issue #6982: PreResponseAuthorizationCheckFilter error
when loading coordinator console
URL: https://github.com/apache/incubator-druid/issues/6982
in 0.13.0, when loading the coordinator console, the following error occurs
(although the subsequent redirect works and authorization checks are performed
correctly in the actual console).
```
ERROR [qtp2058526846-335]
io.druid.server.security.PreResponseAuthorizationCheckFilter - Request did not
have an authorization check performed.:
{class=io.druid.server.security.PreResponseAuthorizationCheckFilter, uri=/,
method=GET, remoteAddr=1.1.1.1 remoteHost=1.1.1.1}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
jon-wei commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253196296 ## File path: docs/content/querying/sql.md ## @@ -119,7 +119,11 @@ Only the COUNT aggregation can accept DISTINCT. |`MAX(expr)`|Takes the maximum of numbers.| |`AVG(expr)`|Averages numbers.| |`APPROX_COUNT_DISTINCT(expr)`|Counts distinct values of expr, which can be a regular column or a hyperUnique column. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`.| -|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or approxHistogram exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_HLL(expr, [lgK, tgtHllType])`|Counts distinct values of expr, which can be a regular column or an [HLL sketch](../development/extensions-core/datasketches-hll.html) column. The `lgK` and `tgtHllType` parameters are described in the HLL sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])`|Counts distinct values of expr, which can be a regular column or a [Theta sketch](../development/extensions-core/datasketches-theta.html) column. The `size` parameter is described in the Theta sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or [approxHistogram](../development/extensions-core/approximate-histograms.html#approximate-histogram-aggregator) exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_QUANTILE_DS(expr, probability, [k])`|Computes approximate quantiles on numeric or [Quantiles sketch](../development/extensions-core/datasketches-quantiles.html) exprs. The "probability" should be between 0 and 1 (exclusive). The `k` parameter is described in the Quantiles sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| Review comment: Whoops, removed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jon-wei commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
jon-wei commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253196340 ## File path: docs/content/querying/sql.md ## @@ -119,7 +119,11 @@ Only the COUNT aggregation can accept DISTINCT. |`MAX(expr)`|Takes the maximum of numbers.| |`AVG(expr)`|Averages numbers.| |`APPROX_COUNT_DISTINCT(expr)`|Counts distinct values of expr, which can be a regular column or a hyperUnique column. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`.| -|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or approxHistogram exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_HLL(expr, [lgK, tgtHllType])`|Counts distinct values of expr, which can be a regular column or an [HLL sketch](../development/extensions-core/datasketches-hll.html) column. The `lgK` and `tgtHllType` parameters are described in the HLL sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])`|Counts distinct values of expr, which can be a regular column or a [Theta sketch](../development/extensions-core/datasketches-theta.html) column. The `size` parameter is described in the Theta sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or [approxHistogram](../development/extensions-core/approximate-histograms.html#approximate-histogram-aggregator) exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_QUANTILE_DS(expr, probability, [k])`|Computes approximate quantiles on numeric or [Quantiles sketch](../development/extensions-core/datasketches-quantiles.html) exprs. The "probability" should be between 0 and 1 (exclusive). The `k` parameter is described in the Quantiles sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_QUANTILE_FIXED_BUCKETS(expr, probability, numBuckets, lowerLimit, upperLimit, [outlierHandlingMode])`|Computes approximate quantiles on numeric or [fixed buckets histogram](../development/extensions-core/approximate-histograms.html#fixed-buckets-histogram) exprs. The "probability" should be between 0 and 1 (exclusive). The `numBuckets`, `lowerLimit`, `upperLimit`, and `outlierHandlingMode` parameters are described in the fixed buckets histogram documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| Review comment: same, removed This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253188129
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.concurrent.LifecycleLock;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final boolean isCacheEnabled;
+ private final ConcurrentMap publishedSegments;
+ private final ScheduledExecutorService scheduledExec;
+ private final long pollPeriodinMS;
+ private LifecycleLock lifecycleLock = new LifecycleLock();
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+Preconditions.checkNotNull(plannerConfig, "plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+this.isCacheEnabled = plannerConfig.isMetadataSegmentCacheEnable();
+this.pollPeriodinMS = plannerConfig.getMetadataSegmentPollPeriod();
+this.publishedSegments = isCacheEnabled ? new ConcurrentHashMap<>(1000) :
null;
+this.scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (!lifecycleLock.canStart()) {
+ throw new ISE("can't start.");
+}
+try {
+ if (isCacheEnabled) {
+scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+lifecycleLock.started();
+log.info("MetadataSegmentView Started.");
+ }
+}
+
[GitHub] vogievetsky commented on issue #6981: Fix node path for building the unified console
vogievetsky commented on issue #6981: Fix node path for building the unified console URL: https://github.com/apache/incubator-druid/pull/6981#issuecomment-459851811 looks good to me, thank you for fixing that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on issue #6980: With latest version of druid is it possible to use OpenStack Swift custom implementation (not Amazon S3)?
jihoonson commented on issue #6980: With latest version of druid is it possible to use OpenStack Swift custom implementation (not Amazon S3)? URL: https://github.com/apache/incubator-druid/issues/6980#issuecomment-459837410 If you want to use `http`, you may need https://github.com/apache/incubator-druid/pull/6954. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson opened a new pull request #6981: Fix node path for building the unified console
jihoonson opened a new pull request #6981: Fix node path for building the unified console URL: https://github.com/apache/incubator-druid/pull/6981 Maven installs a particular version of `node` by itself in the `generate-resources` phase, and is supposed to use the installed `node` for subsequent build. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] anantmf opened a new issue #6980: With latest version of druid is it possible to use OpenStack Swift custom implementation (not Amazon S3)?
anantmf opened a new issue #6980: With latest version of druid is it possible to use OpenStack Swift custom implementation (not Amazon S3)? URL: https://github.com/apache/incubator-druid/issues/6980 With latest version of druid is it possible to use OpenStack Swift custom implementation (not Amazon S3)? I am having some issue in setting up this, so wanted to check if this is supported or not before asking further questions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253159655
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
That said, there are so many different patterns in use for `start` and
`stop` idk what is the _right_ thing to do
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please conta
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253159045
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
I _didn't_ think lifecycleLock is for start and stop but for things that
live lifetime of service, like `HttpServerInventoryView` uses a lifecycle lock.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253155145
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
Ah, I think those locks are required for some modules which can be started
and stopped repeatedly, like whenever getting/losing the zk leadership. For
them, start() and stop() can be called simultaneously at any time, so they need
a lock for coordination.
However, `MetadataSegmentView` has the same lifecycle with the broker. Once
the broker is stopped, `MetadataSegmentView.start
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253154261
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
Moved `scheduledExec` initialization to constructor. May be should add a
`lock` around start and stop ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infra
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253150936
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
hmm... I don't remember off the top of my head if it is _required_, but many
other lifecycle annotated methods have such synchronization around start and
stop, especially if they are scheduling threads.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use t
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253137580
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.concurrent.LifecycleLock;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final boolean isCacheEnabled;
+ private final ConcurrentMap publishedSegments;
+ private ScheduledExecutorService scheduledExec;
+ private final long pollPeriodinMS;
+ private LifecycleLock lifecycleLock = new LifecycleLock();
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+Preconditions.checkNotNull(plannerConfig, "plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+this.isCacheEnabled = plannerConfig.isMetadataSegmentCacheEnable();
+this.pollPeriodinMS = plannerConfig.getMetadataSegmentPollPeriod();
+this.publishedSegments = isCacheEnabled ? new ConcurrentHashMap<>(1000) :
new ConcurrentHashMap<>();
Review comment:
hmm, may be
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
[GitHub] clintropolis commented on a change in pull request #6971: Added checkstyle for "Methods starting with Capital Letters"
clintropolis commented on a change in pull request #6971: Added checkstyle for
"Methods starting with Capital Letters"
URL: https://github.com/apache/incubator-druid/pull/6971#discussion_r253149585
##
File path: sql/src/test/java/org/apache/druid/sql/calcite/CalciteQueryTest.java
##
@@ -133,15 +133,15 @@ public void testSelectCountStart() throws Exception
CalciteTests.REGULAR_USER_AUTH_RESULT,
ImmutableList.of(Druids.newTimeseriesQueryBuilder()
.dataSource(CalciteTests.DATASOURCE1)
- .intervals(QSS(Filtration.eternity()))
- .filters(SELECTOR("dim2", "0", null))
+ .intervals(qss(Filtration.eternity()))
Review comment:
Hmm, so I think I actually prefer these test shorthand methods as ALLCAPS
instead of lowercase, makes it to parse for me at least. Was the intention to
disallow all uppercase static methods too or just methods that started with an
uppercase letter and were not fully uppercase?
If no one agrees with me feel free to disregard this review 🙃
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] hpandeycodeit opened a new pull request #6979: Added IntelliJ Inspection
hpandeycodeit opened a new pull request #6979: Added IntelliJ Inspection URL: https://github.com/apache/incubator-druid/pull/6979 Issue #6856 Make IntelliJ inspection "Iteration over keySet() could be replaced with entrySet()" a error This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253134452
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.concurrent.LifecycleLock;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.ISE;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final boolean isCacheEnabled;
+ private final ConcurrentMap publishedSegments;
+ private ScheduledExecutorService scheduledExec;
+ private final long pollPeriodinMS;
+ private LifecycleLock lifecycleLock = new LifecycleLock();
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+Preconditions.checkNotNull(plannerConfig, "plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+this.isCacheEnabled = plannerConfig.isMetadataSegmentCacheEnable();
+this.pollPeriodinMS = plannerConfig.getMetadataSegmentPollPeriod();
+this.publishedSegments = isCacheEnabled ? new ConcurrentHashMap<>(1000) :
new ConcurrentHashMap<>();
Review comment:
null ifCacheEnabled = false?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253132485
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
Why synchronization?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
---
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253133403
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
I think it can happen, but it would be better to initialize `scheduledExec`
in the constructor and make it final rather than using synchronization.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries a
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253134211
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
Review comment:
An nvm. Github shows an old version..
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253133696
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
Review comment:
@surekhasaharan hmm I don't see a config for this in `PlannerConfig`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] jihoonson commented on a change in pull request #6901: Introduce published segment cache in broker
jihoonson commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253132922
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
Do you mean, stop() can be called before `scheduledExec` is set?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With r
[GitHub] hpandeycodeit edited a comment on issue #6936: Checkstyle: prohibit method names starting with a capital letter
hpandeycodeit edited a comment on issue #6936: Checkstyle: prohibit method names starting with a capital letter URL: https://github.com/apache/incubator-druid/issues/6936#issuecomment-459784938 @leventov Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] hpandeycodeit commented on issue #6936: Checkstyle: prohibit method names starting with a capital letter
hpandeycodeit commented on issue #6936: Checkstyle: prohibit method names starting with a capital letter URL: https://github.com/apache/incubator-druid/issues/6936#issuecomment-459784938 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253100162
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
Review comment:
yeah, makes sense
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253078258
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerConfig.java
##
@@ -66,6 +66,14 @@
@JsonProperty
private DateTimeZone sqlTimeZone = DateTimeZone.UTC;
+ @JsonProperty
+ private boolean metadataSegmentCacheEnable = false;
+
+ public boolean isMetadataSegmentCacheEnable()
+ {
+return metadataSegmentCacheEnable;
Review comment:
added
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253076594
##
File path:
server/src/main/java/org/apache/druid/client/DataSegmentInterner.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.google.common.collect.Interner;
+import com.google.common.collect.Interners;
+import org.apache.druid.timeline.DataSegment;
+
+/**
+ * Interns the DataSegment object in order to share the reference for same
DataSegment.
+ * It uses two separate interners for realtime and historical segments to
prevent
+ * overwriting the size of a segment which was served by a historical and
later served
+ * by another realtime server, since realtime server always publishes with
size 0.
+ */
+public class DataSegmentInterner
+{
+ private static final Interner REALTIME_INTERNER =
Interners.newWeakInterner();
+ private static final Interner HISTORICAL_INTERNER =
Interners.newWeakInterner();
+
+ private DataSegmentInterner()
+ {
+
+ }
+
+ public static DataSegment intern(DataSegment segment)
+ {
+// A segment learns it's size and dimensions when it moves from a relatime
to historical server
+// for that reason, we are using it's size as the indicator to decide
whether to use REALTIME or
+// HISTORICAL interner.
+return segment.getSize() > 0 ? HISTORICAL_INTERNER.intern(segment) :
REALTIME_INTERNER.intern(segment);
+ }
+
Review comment:
removed empty line here
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253102501
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
Review comment:
added the `LifecycleLock`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253102825
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
added cache enabled check and `LifecycleLock` here as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With re
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253074570
##
File path:
server/src/main/java/org/apache/druid/client/DataSegmentInterner.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.google.common.collect.Interner;
+import com.google.common.collect.Interners;
+import org.apache.druid.timeline.DataSegment;
+
+/**
+ * Interns the DataSegment object in order to share the reference for same
DataSegment.
+ * It uses two separate interners for realtime and historical segments to
prevent
+ * overwriting the size of a segment which was served by a historical and
later served
+ * by another realtime server, since realtime server always publishes with
size 0.
+ */
+public class DataSegmentInterner
+{
+ private static final Interner REALTIME_INTERNER =
Interners.newWeakInterner();
+ private static final Interner HISTORICAL_INTERNER =
Interners.newWeakInterner();
+
+ private DataSegmentInterner()
+ {
+
Review comment:
added the comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] surekhasaharan commented on a change in pull request #6901: Introduce published segment cache in broker
surekhasaharan commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r25304
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
Review comment:
yeah, added a config in `PlannerConfig` for this as well. @jihoonson also
asked for this before i think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov commented on issue #6936: Checkstyle: prohibit method names starting with a capital letter
leventov commented on issue #6936: Checkstyle: prohibit method names starting with a capital letter URL: https://github.com/apache/incubator-druid/issues/6936#issuecomment-459718436 @hpandeycodeit it seems fine - two automatic checks passed. Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov opened a new issue #6978: Future not cancelled if interrupted in MemcachedCache
leventov opened a new issue #6978: Future not cancelled if interrupted in MemcachedCache URL: https://github.com/apache/incubator-druid/issues/6978 See [checklist](https://medium.com/@leventov/code-review-checklist-java-concurrency-49398c326154), item 11.3. Similar problems could exist in other places in the codebase. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
clintropolis commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253018957 ## File path: docs/content/querying/sql.md ## @@ -119,7 +119,11 @@ Only the COUNT aggregation can accept DISTINCT. |`MAX(expr)`|Takes the maximum of numbers.| |`AVG(expr)`|Averages numbers.| |`APPROX_COUNT_DISTINCT(expr)`|Counts distinct values of expr, which can be a regular column or a hyperUnique column. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`.| -|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or approxHistogram exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_HLL(expr, [lgK, tgtHllType])`|Counts distinct values of expr, which can be a regular column or an [HLL sketch](../development/extensions-core/datasketches-hll.html) column. The `lgK` and `tgtHllType` parameters are described in the HLL sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])`|Counts distinct values of expr, which can be a regular column or a [Theta sketch](../development/extensions-core/datasketches-theta.html) column. The `size` parameter is described in the Theta sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or [approxHistogram](../development/extensions-core/approximate-histograms.html#approximate-histogram-aggregator) exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_QUANTILE_DS(expr, probability, [k])`|Computes approximate quantiles on numeric or [Quantiles sketch](../development/extensions-core/datasketches-quantiles.html) exprs. The "probability" should be between 0 and 1 (exclusive). The `k` parameter is described in the Quantiles sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| Review comment: Is the `useApproximateCountDistinct` comment relevant here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL
clintropolis commented on a change in pull request #6951: Add more sketch aggregator support in Druid SQL URL: https://github.com/apache/incubator-druid/pull/6951#discussion_r253019025 ## File path: docs/content/querying/sql.md ## @@ -119,7 +119,11 @@ Only the COUNT aggregation can accept DISTINCT. |`MAX(expr)`|Takes the maximum of numbers.| |`AVG(expr)`|Averages numbers.| |`APPROX_COUNT_DISTINCT(expr)`|Counts distinct values of expr, which can be a regular column or a hyperUnique column. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`.| -|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or approxHistogram exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_HLL(expr, [lgK, tgtHllType])`|Counts distinct values of expr, which can be a regular column or an [HLL sketch](../development/extensions-core/datasketches-hll.html) column. The `lgK` and `tgtHllType` parameters are described in the HLL sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_COUNT_DISTINCT_DS_THETA(expr, [size])`|Counts distinct values of expr, which can be a regular column or a [Theta sketch](../development/extensions-core/datasketches-theta.html) column. The `size` parameter is described in the Theta sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_QUANTILE(expr, probability, [resolution])`|Computes approximate quantiles on numeric or [approxHistogram](../development/extensions-core/approximate-histograms.html#approximate-histogram-aggregator) exprs. The "probability" should be between 0 and 1 (exclusive). The "resolution" is the number of centroids to use for the computation. Higher resolutions will give more precise results but also have higher overhead. If not provided, the default resolution is 50. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| +|`APPROX_QUANTILE_DS(expr, probability, [k])`|Computes approximate quantiles on numeric or [Quantiles sketch](../development/extensions-core/datasketches-quantiles.html) exprs. The "probability" should be between 0 and 1 (exclusive). The `k` parameter is described in the Quantiles sketch documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [DataSketches extension](../development/extensions-core/datasketches-extensions.html) must be loaded to use this function.| +|`APPROX_QUANTILE_FIXED_BUCKETS(expr, probability, numBuckets, lowerLimit, upperLimit, [outlierHandlingMode])`|Computes approximate quantiles on numeric or [fixed buckets histogram](../development/extensions-core/approximate-histograms.html#fixed-buckets-histogram) exprs. The "probability" should be between 0 and 1 (exclusive). The `numBuckets`, `lowerLimit`, `upperLimit`, and `outlierHandlingMode` parameters are described in the fixed buckets histogram documentation. This is always approximate, regardless of the value of "useApproximateCountDistinct". See also `COUNT(DISTINCT expr)`. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.| Review comment: ditto `useApproximateCountDistinct` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov opened a new issue #6977: Design of MaterializedViewQuery
leventov opened a new issue #6977: Design of MaterializedViewQuery URL: https://github.com/apache/incubator-druid/issues/6977 Currently, #5556 implements it as `MaterializedViewQuery` and `MaterializedViewQueryQueryToolChest`, i. e. in the same type framework as `Query` and `QueryToolChest`. However, `MaterializedViewQuery` and `MaterializedViewQueryQueryToolChest` are shallow and pass-through classes. `MaterializedViewQueryQueryToolChest` couldn't sensibly return something from `getResultTypeReference()`, so I wonder why it didn't throw NPE in `DirectDruidClient` before, but I hit this problem in my change #6898. There should probably be an abstraction like "query modifier" and other parts of the infrastructure should recognize it. cc @zhangxinyu1 @jihoonson This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6950: bloom filter sql aggregator
clintropolis commented on a change in pull request #6950: bloom filter sql
aggregator
URL: https://github.com/apache/incubator-druid/pull/6950#discussion_r253018282
##
File path:
extensions-core/druid-bloom-filter/src/main/java/org/apache/druid/query/aggregation/bloom/sql/BloomFilterSqlAggregator.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.query.aggregation.bloom.sql;
+
+import org.apache.calcite.rel.core.AggregateCall;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlAggFunction;
+import org.apache.calcite.sql.SqlFunctionCategory;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.type.OperandTypes;
+import org.apache.calcite.sql.type.ReturnTypes;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.calcite.sql.type.SqlTypeName;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.query.aggregation.AggregatorFactory;
+import org.apache.druid.query.aggregation.bloom.BloomFilterAggregatorFactory;
+import org.apache.druid.query.dimension.DefaultDimensionSpec;
+import org.apache.druid.query.dimension.DimensionSpec;
+import org.apache.druid.query.dimension.ExtractionDimensionSpec;
+import org.apache.druid.segment.VirtualColumn;
+import org.apache.druid.segment.column.ValueType;
+import org.apache.druid.segment.virtual.ExpressionVirtualColumn;
+import org.apache.druid.sql.calcite.aggregation.Aggregation;
+import org.apache.druid.sql.calcite.aggregation.SqlAggregator;
+import org.apache.druid.sql.calcite.expression.DruidExpression;
+import org.apache.druid.sql.calcite.expression.Expressions;
+import org.apache.druid.sql.calcite.planner.Calcites;
+import org.apache.druid.sql.calcite.planner.PlannerContext;
+import org.apache.druid.sql.calcite.table.RowSignature;
+
+import javax.annotation.Nullable;
+import java.util.ArrayList;
+import java.util.List;
+
+public class BloomFilterSqlAggregator implements SqlAggregator
+{
+ private static final SqlAggFunction FUNCTION_INSTANCE = new
BloomFilterSqlAggFunction();
+ private static final String NAME = "BLOOM_FILTER";
+
+ @Override
+ public SqlAggFunction calciteFunction()
+ {
+return FUNCTION_INSTANCE;
+ }
+
+ @Nullable
+ @Override
+ public Aggregation toDruidAggregation(
+ PlannerContext plannerContext,
+ RowSignature rowSignature,
+ RexBuilder rexBuilder,
+ String name,
+ AggregateCall aggregateCall,
+ Project project,
+ List existingAggregations,
+ boolean finalizeAggregations
+ )
+ {
+final RexNode inputOperand = Expressions.fromFieldAccess(
+rowSignature,
+project,
+aggregateCall.getArgList().get(0)
+);
+final DruidExpression input = Expressions.toDruidExpression(
+plannerContext,
+rowSignature,
+inputOperand
+);
+if (input == null) {
+ return null;
+}
+
+final AggregatorFactory aggregatorFactory;
+final String aggName = StringUtils.format("%s:agg", name);
+final RexNode maxNumEntriesOperand = Expressions.fromFieldAccess(
+rowSignature,
+project,
+aggregateCall.getArgList().get(1)
+);
+
+if (!maxNumEntriesOperand.isA(SqlKind.LITERAL)) {
+ // Probability must be a literal in order to plan.
Review comment:
🍝
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov commented on a change in pull request #6948: Add guava compatability up to 27.0.1
leventov commented on a change in pull request #6948: Add guava compatability up to 27.0.1 URL: https://github.com/apache/incubator-druid/pull/6948#discussion_r253013521 ## File path: core/src/main/java/org/apache/druid/common/guava/GuavaUtils.java ## @@ -62,4 +100,70 @@ public static Long tryParseLong(@Nullable String string) return null; } + + /** + * Try the various methods (zero arguments) against the object. This is handy for maintaining guava compatability + * + * @param object The object to call the methods on + * @param methods The sequence of methods to call + * @param The return type + * + * @return The result of invoking the method on the object + */ + private static T tryMethods(Object object, Class assignableTo, String... methods) Review comment: Documentation should reference the specific commit and the range of lines (as a Github link) in Guava from where this was copied. Same for other non-trivial parts of the copied code. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253012385
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
Review comment:
This should probably only be allocated with this size if cache is enabled
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r252971250
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/planner/PlannerConfig.java
##
@@ -66,6 +66,14 @@
@JsonProperty
private DateTimeZone sqlTimeZone = DateTimeZone.UTC;
+ @JsonProperty
+ private boolean metadataSegmentCacheEnable = false;
+
+ public boolean isMetadataSegmentCacheEnable()
+ {
+return metadataSegmentCacheEnable;
Review comment:
Please add to `equals`, `toString`, `hashCode` methods
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r252971552
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
+ scheduledExec.schedule(new PollTask(), 0, TimeUnit.MILLISECONDS);
+}
+ }
+
+ @LifecycleStop
+ public void stop()
+ {
+scheduledExec.shutdownNow();
Review comment:
This should only stop if cache is enabled, and also synchronization i think
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apach
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253008712
##
File path:
server/src/main/java/org/apache/druid/client/DataSegmentInterner.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.google.common.collect.Interner;
+import com.google.common.collect.Interners;
+import org.apache.druid.timeline.DataSegment;
+
+/**
+ * Interns the DataSegment object in order to share the reference for same
DataSegment.
+ * It uses two separate interners for realtime and historical segments to
prevent
+ * overwriting the size of a segment which was served by a historical and
later served
+ * by another realtime server, since realtime server always publishes with
size 0.
+ */
+public class DataSegmentInterner
+{
+ private static final Interner REALTIME_INTERNER =
Interners.newWeakInterner();
+ private static final Interner HISTORICAL_INTERNER =
Interners.newWeakInterner();
+
+ private DataSegmentInterner()
+ {
+
+ }
+
+ public static DataSegment intern(DataSegment segment)
+ {
+// A segment learns it's size and dimensions when it moves from a relatime
to historical server
+// for that reason, we are using it's size as the indicator to decide
whether to use REALTIME or
+// HISTORICAL interner.
+return segment.getSize() > 0 ? HISTORICAL_INTERNER.intern(segment) :
REALTIME_INTERNER.intern(segment);
+ }
+
Review comment:
nit: extra space (there are a few others scattered about as well, please
clean up if you have the chance)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253008613
##
File path:
server/src/main/java/org/apache/druid/client/DataSegmentInterner.java
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.client;
+
+import com.google.common.collect.Interner;
+import com.google.common.collect.Interners;
+import org.apache.druid.timeline.DataSegment;
+
+/**
+ * Interns the DataSegment object in order to share the reference for same
DataSegment.
+ * It uses two separate interners for realtime and historical segments to
prevent
+ * overwriting the size of a segment which was served by a historical and
later served
+ * by another realtime server, since realtime server always publishes with
size 0.
+ */
+public class DataSegmentInterner
+{
+ private static final Interner REALTIME_INTERNER =
Interners.newWeakInterner();
+ private static final Interner HISTORICAL_INTERNER =
Interners.newWeakInterner();
+
+ private DataSegmentInterner()
+ {
+
Review comment:
please add comment `// No instantiation.`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r252972457
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
+ private static final EmittingLogger log = new
EmittingLogger(MetadataSegmentView.class);
+
+ private final DruidLeaderClient coordinatorDruidLeaderClient;
+ private final ObjectMapper jsonMapper;
+ private final BytesAccumulatingResponseHandler responseHandler;
+ private final BrokerSegmentWatcherConfig segmentWatcherConfig;
+
+ private final ConcurrentMap publishedSegments = new
ConcurrentHashMap<>(1000);
+ private ScheduledExecutorService scheduledExec;
+ final PlannerConfig plannerConfig;
+
+ @Inject
+ public MetadataSegmentView(
+ final @Coordinator DruidLeaderClient druidLeaderClient,
+ final ObjectMapper jsonMapper,
+ final BytesAccumulatingResponseHandler responseHandler,
+ final BrokerSegmentWatcherConfig segmentWatcherConfig,
+ final PlannerConfig plannerConfig
+ )
+ {
+this.plannerConfig = Preconditions.checkNotNull(plannerConfig,
"plannerConfig");
+this.coordinatorDruidLeaderClient = druidLeaderClient;
+this.jsonMapper = jsonMapper;
+this.responseHandler = responseHandler;
+this.segmentWatcherConfig = segmentWatcherConfig;
+ }
+
+ @LifecycleStart
+ public void start()
+ {
+if (plannerConfig.isMetadataSegmentCacheEnable()) {
+ scheduledExec =
Execs.scheduledSingleThreaded("MetadataSegmentView-Cache--%d");
Review comment:
I think this needs some sort of synchronization and started check, maybe a
`LifecycleLock` if it should start and stop with the service, or maybe just
`lock` and `AtomicBoolean` if it can stop and start while service is running.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Wit
[GitHub] clintropolis commented on a change in pull request #6901: Introduce published segment cache in broker
clintropolis commented on a change in pull request #6901: Introduce published
segment cache in broker
URL: https://github.com/apache/incubator-druid/pull/6901#discussion_r253005916
##
File path:
sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java
##
@@ -0,0 +1,221 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.sql.calcite.schema;
+
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.JavaType;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Preconditions;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.inject.Inject;
+import org.apache.druid.client.BrokerSegmentWatcherConfig;
+import org.apache.druid.client.DataSegmentInterner;
+import org.apache.druid.client.JsonParserIterator;
+import org.apache.druid.client.coordinator.Coordinator;
+import org.apache.druid.discovery.DruidLeaderClient;
+import org.apache.druid.guice.ManageLifecycle;
+import org.apache.druid.java.util.common.DateTimes;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.concurrent.Execs;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStart;
+import org.apache.druid.java.util.common.lifecycle.LifecycleStop;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+import org.apache.druid.java.util.http.client.Request;
+import org.apache.druid.server.coordinator.BytesAccumulatingResponseHandler;
+import org.apache.druid.sql.calcite.planner.PlannerConfig;
+import org.apache.druid.timeline.DataSegment;
+import org.jboss.netty.handler.codec.http.HttpMethod;
+import org.joda.time.DateTime;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Iterator;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * This class polls the coordinator in background to keep the latest published
segments.
+ * Provides {@link #getPublishedSegments()} for others to get segments in
metadata store.
+ */
+@ManageLifecycle
+public class MetadataSegmentView
+{
+
+ private static final long POLL_PERIOD_IN_MS = 6;
Review comment:
Can this be configurable too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] leventov commented on issue #6942: Shade Guava manually
leventov commented on issue #6942: Shade Guava manually URL: https://github.com/apache/incubator-druid/issues/6942#issuecomment-459687733 To me it seems that there is a problem with Charles's approach (in #6948 and #6815) (also we have `Closer` and maybe some other classes copied from Guava already): this is very ad-hoc, each class is copied independently. There is no list of them all. Each class should be updated in separation. The likely result is that we probably won't update them. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] asdf2014 merged pull request #6975: use System.err and System.out to print exit messages on CliPeon
asdf2014 merged pull request #6975: use System.err and System.out to print exit messages on CliPeon URL: https://github.com/apache/incubator-druid/pull/6975 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: use System.err and System.out to print exit messages on CliPeon (#6975)
This is an automated email from the ASF dual-hosted git repository.
asdf2014 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 5c0fbbd use System.err and System.out to print exit messages on
CliPeon (#6975)
5c0fbbd is described below
commit 5c0fbbda1ba2f0241b18d38b5ca62553fb37facb
Author: Clint Wylie
AuthorDate: Fri Feb 1 02:54:14 2019 -0800
use System.err and System.out to print exit messages on CliPeon (#6975)
* use System.err and System.out to print exit messages on CliPeon
* more
* not necessarily a stopping error...
---
services/src/main/java/org/apache/druid/cli/CliPeon.java | 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/services/src/main/java/org/apache/druid/cli/CliPeon.java
b/services/src/main/java/org/apache/druid/cli/CliPeon.java
index 991429b..71dc305 100644
--- a/services/src/main/java/org/apache/druid/cli/CliPeon.java
+++ b/services/src/main/java/org/apache/druid/cli/CliPeon.java
@@ -35,6 +35,7 @@ import com.google.inject.name.Names;
import io.airlift.airline.Arguments;
import io.airlift.airline.Command;
import io.airlift.airline.Option;
+import io.netty.util.SuppressForbidden;
import org.apache.druid.client.cache.CacheConfig;
import org.apache.druid.client.coordinator.CoordinatorClient;
import org.apache.druid.client.indexing.HttpIndexingServiceClient;
@@ -345,6 +346,7 @@ public class CliPeon extends GuiceRunnable
);
}
+ @SuppressForbidden(reason = "System#out, System#err")
@Override
public void run()
{
@@ -375,14 +377,15 @@ public class CliPeon extends GuiceRunnable
Runtime.getRuntime().removeShutdownHook(hook);
}
catch (IllegalStateException e) {
- log.warn("Cannot remove shutdown hook, already shutting down");
+ System.err.println("Cannot remove shutdown hook, already shutting
down!");
}
}
catch (Throwable t) {
-log.error(t, "Error when starting up. Failing.");
+System.err.println("Error!");
+System.err.println(Throwables.getStackTraceAsString(t));
System.exit(1);
}
- log.info("Finished peon task");
+ System.out.println("Finished peon task");
}
catch (Exception e) {
throw Throwables.propagate(e);
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] lxqfy opened a new pull request #6976: Updated metrics.md to fix format
lxqfy opened a new pull request #6976: Updated metrics.md to fix format URL: https://github.com/apache/incubator-druid/pull/6976 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] mihai-cazacu-adswizz edited a comment on issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service
mihai-cazacu-adswizz edited a comment on issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service URL: https://github.com/apache/incubator-druid/issues/6945#issuecomment-459649812 > The timeline of derivativeDataSource is expected to be the same with baseDataSource's. Your can get the timeline of a dataSource from coordinator UI. Inspecting the timeline, the derivative data source has one day missing (the last day of the month). The base data source:  and the derivative one:  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] mihai-cazacu-adswizz commented on issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service
mihai-cazacu-adswizz commented on issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service URL: https://github.com/apache/incubator-druid/issues/6945#issuecomment-459649812 > The timeline of derivativeDataSource is expected to be the same with baseDataSource's. Your can get the timeline of a dataSource from coordinator UI. Inspecting the timeline, the derivative base data source has one day missing (the last day of the month). The base data source:  and the derivative one:  This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] mihai-cazacu-adswizz commented on issue #6969: The materialized view returns different results than the base data source
mihai-cazacu-adswizz commented on issue #6969: The materialized view returns different results than the base data source URL: https://github.com/apache/incubator-druid/issues/6969#issuecomment-459645304 I see. Thank you so much for your help! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] mihai-cazacu-adswizz closed issue #6969: The materialized view returns different results than the base data source
mihai-cazacu-adswizz closed issue #6969: The materialized view returns different results than the base data source URL: https://github.com/apache/incubator-druid/issues/6969 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] mihai-cazacu-adswizz closed issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service
mihai-cazacu-adswizz closed issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service URL: https://github.com/apache/incubator-druid/issues/6945 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] mihai-cazacu-adswizz commented on issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service
mihai-cazacu-adswizz commented on issue #6945: The materialized view is not created for a data source produced by Kafka Indexing Service URL: https://github.com/apache/incubator-druid/issues/6945#issuecomment-459644475 > When you submited the derivativeDataSource task, were there any segments of base dataSource you can see from coordinator web UI? Yes. there were ingested one month of data. The coordinator shows this (for the base data source): `207 segments in 22 intervals`. ``` 2019-01-31T112 137 kB 2019-01-31T103 119 kB 2019-01-31T071 49.8 kB 2019-01-31T066 273 kB 2019-01-30T169 381 kB 2019-01-30T1510 380 kB 2019-01-30T142 162 kB 2019-01-30T135 182 kB 2019-01-30T1221 900 kB 2019-01-30T1116 644 kB 2019-01-30T103 116 kB 2019-01-30T098 366 kB 2019-01-30T085 20.4 MB 2019-01-30T0729 128 MB 2019-01-30T061 68.2 kB 2019-01-29T1622 994 kB 2019-01-29T151 47.9 kB 2019-01-29T1414 671 kB 2019-01-29T137 313 kB 2019-01-29T123 116 kB 2019-01-29T1117 30.6 MB 2019-01-29T1022 37 MB ``` As I said [before](https://github.com/apache/incubator-druid/issues/6945#issuecomment-459258335), I can create the view now but it's strange that, using the same spec, a few days ago I wasn't able to create that view. I'll close the issue because I cannot reproduce it anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] hpandeycodeit commented on issue #6936: Checkstyle: prohibit method names starting with a capital letter
hpandeycodeit commented on issue #6936: Checkstyle: prohibit method names starting with a capital letter URL: https://github.com/apache/incubator-druid/issues/6936#issuecomment-459642819 @leventov I am seeing some Inspection/Teamcity build failing. I don't see any errors in my local build though. Any idea? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
