[GitHub] [incubator-druid] wengwh opened a new issue #8684: i develop a lookup extension,read hbase into lookup

2019-10-16 Thread GitBox
wengwh opened a new issue #8684: i develop a lookup extension,read hbase into 
lookup 
URL: https://github.com/apache/incubator-druid/issues/8684
 
 
   i develop a lookup extension,read hbase into lookup, i deploy success,data 
load success, 
   
   but i load datasource are fail 
   
   druid version:0.15.1
   
   i put hbase-client.jar into druid/extensions/myextension/
   but load datasource is fail,the msg:not found class。
   
   ERROR:
   2019-10-16T12:52:13,833 INFO [task-runner-0-priority-0] 
org.apache.hadoop.mapreduce.Job - Task Id : 
attempt_1569862598380_177757_m_00_0, Status : FAILED
   Error: java.lang.ClassNotFoundException: 
org.apache.hadoop.hbase.client.Result
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at 
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:202)
at 
org.apache.hadoop.util.ApplicationClassLoader.loadClass(ApplicationClassLoader.java:170)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975)
at 
com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
at 
com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:380)
at 
com.google.inject.internal.ConstructorBindingImpl.getInternalDependencies(ConstructorBindingImpl.java:165)
at 
com.google.inject.internal.InjectorImpl.getInternalDependencies(InjectorImpl.java:616)
at 
com.google.inject.internal.InjectorImpl.cleanup(InjectorImpl.java:572)
at 
com.google.inject.internal.InjectorImpl.initializeJitBinding(InjectorImpl.java:558)
at 
com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:887)
at 
com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:808)
at 
com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:285)
at 
com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:217)
at 
com.google.inject.internal.InjectorImpl.getInternalFactory(InjectorImpl.java:893)
at com.google.inject.internal.FactoryProxy.notify(FactoryProxy.java:46)
at 
com.google.inject.internal.ProcessedBindingData.runCreationListeners(ProcessedBindingData.java:50)
at 
com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:134)
at 
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:73)
at com.google.inject.Guice.createInjector(Guice.java:62)
at 
org.apache.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:419)
at 
org.apache.druid.indexer.HadoopDruidIndexerConfig.(HadoopDruidIndexerConfig.java:102)
at 
org.apache.druid.indexer.HadoopDruidIndexerMapper.setup(HadoopDruidIndexerMapper.java:53)
at 
org.apache.druid.indexer.DetermineHashedPartitionsJob$DetermineCardinalityMapper.setup(DetermineHashedPartitionsJob.java:269)
at 
org.apache.druid.indexer.DetermineHashedPartitionsJob$DetermineCardinalityMapper.run(DetermineHashedPartitionsJob.java:326)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1707)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-16 Thread GitBox
clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335410227
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-16 Thread GitBox
clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335410227
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] denever commented on issue #8656: Message rejection absolute date

2019-10-16 Thread GitBox
denever commented on issue #8656: Message rejection absolute date
URL: https://github.com/apache/incubator-druid/pull/8656#issuecomment-542636264
 
 
   > thanks, can you also update the necessary documentation and mention the 
precedence of `lateMessageRejectionStartDateTime` vs 
`lateMessageRejectionPeriod` or maybe there should be some check ensuring user 
doesn't specify both as that is most likely an user error.
   
   I've update documentation and mentioned the precedence


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] leventov commented on issue #8558: 7227 : Prohibit Non Final Static Field

2019-10-16 Thread GitBox
leventov commented on issue #8558: 7227 : Prohibit Non Final Static Field
URL: https://github.com/apache/incubator-druid/pull/8558#issuecomment-542562851
 
 
   @SandishKumarHN can you see these errors yourself 
[here](https://teamcity.jetbrains.com/viewLog.html?buildId=2590045=Inspection=OpenSourceProjects_Druid_InspectionsPullRequests),
 or there is some sort of access problem that prevents your (a non-admin user) 
from seeing these problems?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] leventov commented on issue #8558: 7227 : Prohibit Non Final Static Field

2019-10-16 Thread GitBox
leventov commented on issue #8558: 7227 : Prohibit Non Final Static Field
URL: https://github.com/apache/incubator-druid/pull/8558#issuecomment-542562434
 
 
   There are 7 related errors:
   
![image](https://user-images.githubusercontent.com/609240/66897260-4849b600-efff-11e9-8bd9-d77eb3699bad.png)
   
   
https://teamcity.jetbrains.com/viewLog.html?buildId=2590045=Inspection=OpenSourceProjects_Druid_InspectionsPullRequests
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jon-wei merged pull request #8671: More Kinesis resharding adjustments

2019-10-16 Thread GitBox
jon-wei merged pull request #8671: More Kinesis resharding adjustments
URL: https://github.com/apache/incubator-druid/pull/8671
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jihoonson closed issue #8489: Stateful auto compaction

2019-10-15 Thread GitBox
jihoonson closed issue #8489: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/issues/8489
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jihoonson merged pull request #8573: Stateful auto compaction

2019-10-15 Thread GitBox
jihoonson merged pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542490718
 
 
   > Can I suggest using something like 
https://metrics.dropwizard.io/3.1.0/manual/healthchecks/?
   
   It looks like something small but useful. We recently started using 
dropwizard in a couple other places and it has been nice.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542489399
 
 
   Great feedback! Thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid-website] vogievetsky merged pull request #40: Autobuild

2019-10-15 Thread GitBox
vogievetsky merged pull request #40: Autobuild
URL: https://github.com/apache/incubator-druid-website/pull/40
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid-website] vogievetsky commented on issue #40: Autobuild

2019-10-15 Thread GitBox
vogievetsky commented on issue #40: Autobuild
URL: 
https://github.com/apache/incubator-druid-website/pull/40#issuecomment-542489172
 
 
   Self merging build website update.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid-website] vogievetsky opened a new pull request #40: Autobuild

2019-10-15 Thread GitBox
vogievetsky opened a new pull request #40: Autobuild
URL: https://github.com/apache/incubator-druid-website/pull/40
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254646
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
 
 Review comment:
   "from the Router"?
   
   Also, "something must be broken" isn't adding much. If we have any 
speculation about why this might happen, add it here, otherwise I think we 
don't have much to say.
   
   Btw, I'd suggest checking for an unauthorized response and translate that 
into something like "You are not authorized to access the XXX API." (for every 
case where the response is generated). This way it'll degrade nicely for secure 
clusters.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335255062
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
+);
+  }
+
+  // Check that the underlying Java is Java 8 the only officially 
supported Java version at the moment.
+  if (
+properties['java.runtime.version'] &&
+!properties['java.runtime.version'].startsWith('1.8')
+  ) {
+controls.addSuggestion(
+  `It looks like are running Java 
${properties['java.runtime.version']}, Druid only officially supports Java 
1.8.x`,
 
 Review comment:
   `java.specification.version` may be easier to parse. (startsWith 1.8 is a 
little brittle, what happens when 1.80 is out )
   
   A period seems more grammatical to me here (vs. a comma).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335256239
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
+);
+  }
+
+  // Check that the underlying Java is Java 8 the only officially 
supported Java version at the moment.
+  if (
+properties['java.runtime.version'] &&
+!properties['java.runtime.version'].startsWith('1.8')
+  ) {
+controls.addSuggestion(
+  `It looks like are running Java 
${properties['java.runtime.version']}, Druid only officially supports Java 
1.8.x`,
+);
+  }
+
+  // Check that "user.timezone"
+  if (properties['user.timezone'] && properties['user.timezone'] !== 
'UTC') {
+controls.addSuggestion(
+  `It looks like "user.timezone" is set to 
${properties['user.timezone']}, it is recommended to set this to "UTC"`,
+);
+  }
+},
+  },
+
+  // -
+  // Coordinator and Overlord
+  // -
+  {
+name: 'Verify the Coordinator and Overlord status',
+check: async controls => {
+  // Make sure that everything in Coordinator's /status is good
+  let myStatus: any;
+  try {
+myStatus = (await axios.get(`/status`)).data;
+  } catch {
+return;
+  }
+
+  let coordinatorStatus: any;
+  try {
+coordinatorStatus = (await 
axios.get(`/proxy/coordinator/status`)).data;
+

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335252556
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
+
+  if (!this.mounted) return;
+  try {
+await check.check({
+  addSuggestion: (message: string) => {
+addToDiagnoses({
+  type: 'suggestion',
+  check: check.name,
+  message,
+});
+  },
+  addIssue: (message: string) => {
+addToDiagnoses({
+  type: 'issue',
+  check: check.name,
+  message,
+});
+  },
+  terminateChecks: () => {
+if (!this.mounted) return;
+this.setState({
+  earlyTermination: `${check.name} early terminated the check 
suite`,
+});
+terminateChecks = true;
+  },
+});
+  } catch (e) {
+addToDiagnoses({
+  type: 'issue',
+  check: check.name,
+  message: `${check.name} encountered an unhandled exception`,
 
 Review comment:
   What might cause this? (Anyone that sees it will be left scratching their 
heads, maybe we can help them out a bit)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254173
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
 
 Review comment:
   How about using "Router" not "cluster" here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335253407
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
+
+  if (!this.mounted) return;
+  try {
+await check.check({
 
 Review comment:
   Does anything get printed for checks that find no issues? Should it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335252958
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
 
 Review comment:
   Is this so they appear slowly and look like they are really working hard?
   
   If so, lol.
   
   Also, I dunno, maybe don't do it. Assuming it's milliseconds, 500ms is a 
long time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335252410
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
+
+  if (!this.mounted) return;
+  try {
+await check.check({
+  addSuggestion: (message: string) => {
+addToDiagnoses({
+  type: 'suggestion',
+  check: check.name,
+  message,
+});
+  },
+  addIssue: (message: string) => {
+addToDiagnoses({
+  type: 'issue',
+  check: check.name,
+  message,
+});
+  },
+  terminateChecks: () => {
+if (!this.mounted) return;
+this.setState({
+  earlyTermination: `${check.name} early terminated the check 
suite`,
 
 Review comment:
   This line made the screenshot confusing to me (I had a moment like, what 
does "Verify own status early terminated" mean? We want to verify that the 
status has early terminated?).
   
   Maybe be more explicit that the check name is part of the message. Also, 
what does it mean when a check early terminated the check suite? Is it because 
the check found a problem? Or because it couldn't run at all? (This should be 
obvious through the message)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335253927
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
 
 Review comment:
   This could be too aggressive. It's fine if different servers have different 
java versions, and would be normal if you're rolling a java upgrade through 
your servers, or if you just simply aren't fastidious about keeping them all at 
exactly the same version. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335255846
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
+);
+  }
+
+  // Check that the underlying Java is Java 8 the only officially 
supported Java version at the moment.
+  if (
+properties['java.runtime.version'] &&
+!properties['java.runtime.version'].startsWith('1.8')
+  ) {
+controls.addSuggestion(
+  `It looks like are running Java 
${properties['java.runtime.version']}, Druid only officially supports Java 
1.8.x`,
+);
+  }
+
+  // Check that "user.timezone"
+  if (properties['user.timezone'] && properties['user.timezone'] !== 
'UTC') {
+controls.addSuggestion(
+  `It looks like "user.timezone" is set to 
${properties['user.timezone']}, it is recommended to set this to "UTC"`,
+);
+  }
+},
+  },
+
+  // -
+  // Coordinator and Overlord
+  // -
+  {
+name: 'Verify the Coordinator and Overlord status',
+check: async controls => {
+  // Make sure that everything in Coordinator's /status is good
+  let myStatus: any;
+  try {
+myStatus = (await axios.get(`/status`)).data;
+  } catch {
+return;
+  }
+
+  let coordinatorStatus: any;
+  try {
+coordinatorStatus = (await 
axios.get(`/proxy/coordinator/status`)).data;
+

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254221
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
 
 Review comment:
   "from the Router"?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

2019-10-15 Thread GitBox
gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254944
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
 
 Review comment:
   I would go with different language, like telling the user that we recommend 
setting this property in order for Coordinator / Overlord APIs to be accessible 
through the Router (and therefore the web console).
   
   The general idea is we should give people some hints about why each check is 
there and what they should do about them if issues are found.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8671: More Kinesis resharding adjustments

2019-10-15 Thread GitBox
jon-wei commented on a change in pull request #8671: More Kinesis resharding 
adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335254796
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1946,53 +1956,152 @@ private boolean updatePartitionDataFromStream()
   }
 }
 
-if (supportsPartitionExpiration()) {
-  // Look for expired shards and remove them from metadata storage and the 
partition groups
-  Set expiredPartitions = new HashSet<>();
-  for (PartitionIdType partitionTd : closedPartitions) {
-if (!partitionIds.contains(partitionTd)) {
-  expiredPartitions.add(partitionTd);
+if (!partitionIds.equals(previousPartitionIds)) {
+  // the set of partition IDs has changed, have any running tasks stop 
early so that we can adjust to the
+  // repartitioning quickly by creating new tasks
+  for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+if (!taskGroup.taskIds().isEmpty()) {
+  // a new partition was added and we are managing active tasks - set 
an early publish time 2 minutes in the
+  // future to give things time to settle
+  earlyStopTime = DateTimes.nowUtc().plusMinutes(2);
 
 Review comment:
   It makes sense to support that for Kafka as well (partitions can be added), 
but I'll do that in a follow-on PR


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8671: More Kinesis resharding adjustments

2019-10-15 Thread GitBox
jon-wei commented on a change in pull request #8671: More Kinesis resharding 
adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335254721
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1946,53 +1956,152 @@ private boolean updatePartitionDataFromStream()
   }
 }
 
-if (supportsPartitionExpiration()) {
-  // Look for expired shards and remove them from metadata storage and the 
partition groups
-  Set expiredPartitions = new HashSet<>();
-  for (PartitionIdType partitionTd : closedPartitions) {
-if (!partitionIds.contains(partitionTd)) {
-  expiredPartitions.add(partitionTd);
+if (!partitionIds.equals(previousPartitionIds)) {
+  // the set of partition IDs has changed, have any running tasks stop 
early so that we can adjust to the
+  // repartitioning quickly by creating new tasks
+  for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+if (!taskGroup.taskIds().isEmpty()) {
+  // a new partition was added and we are managing active tasks - set 
an early publish time 2 minutes in the
+  // future to give things time to settle
+  earlyStopTime = DateTimes.nowUtc().plusMinutes(2);
 
 Review comment:
   I updated this to use a configurable `repartitionTransitionDuration` 
property under the supervisor tuning config, along with docs on what the 
property is for (giving the stream time to write records to the new shards 
before transitioning to a new set of tasks)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jnaous commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
jnaous commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542479729
 
 
   Can I suggest using something like
   https://metrics.dropwizard.io/3.1.0/manual/healthchecks/?
   
   On Tue, Oct 15, 2019 at 6:47 PM Vadim Ogievetsky 
   wrote:
   
   > @himanshug  I 100% agree and I know @gianm
   >  will agree with you also. In fact @gianm
   >  was advocating for the checks should live in
   > Java and I was like: "do I look like James Gosling to you?"
   >
   > But in all seriousness I think you should look at this PR as a working
   > prototype. I want to verify/prove that a 1 button troubleshooting wizard
   > would be a useful addition to the community. I also wanted to have a
   > platform to go around the Druid devs and support ppl that I know to solicit
   > check from them.
   >
   > I think if this proves to be useful then the next step would be to move
   > the majority of the tests into Java and expose them as an endpoint. I would
   > make an issue to track that as soon as this PR is merged.
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
   
   -- 
   Jad Naous
   Imply | VP R
   650-521-3425
   jad.na...@imply.io
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542474187
 
 
   @himanshug I 100% agree and I know @gianm will agree with you also. In fact 
@gianm was advocating for the checks should live in Java and I was like: "do I 
look like James Gosling to you?"
   
   But in all seriousness I think you should look at this PR as a working 
prototype. I want to verify/prove that a 1 button troubleshooting wizard would 
be a useful addition to the community. I also wanted to have a platform to go 
around the Druid devs and support ppl that I know to solicit check from them.
   
   I think if this proves to be useful then the next step would be to move the 
majority of the tests into Java and expose them as an endpoint. I would make an 
issue to track that as soon as this PR is merged.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] SandishKumarHN commented on issue #8558: 7227 : Prohibit Non Final Static Field

2019-10-15 Thread GitBox
SandishKumarHN commented on issue #8558: 7227 : Prohibit Non Final Static Field
URL: https://github.com/apache/incubator-druid/pull/8558#issuecomment-542450809
 
 
   @leventov inspection failure from 
   `
   SeekableStreamSupervisor.java (4)
   175: TaskGroup() Optional used as type for parameter 
'minimumMessageTime'
   `
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8671: More Kinesis resharding adjustments

2019-10-15 Thread GitBox
clintropolis commented on a change in pull request #8671: More Kinesis 
resharding adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335219624
 
 

 ##
 File path: 
extensions-core/kinesis-indexing-service/src/test/java/org/apache/druid/indexing/kinesis/supervisor/KinesisSupervisorTest.java
 ##
 @@ -3885,8 +3906,8 @@ public void testShardSplit() throws Exception
 SHARD_ID0,
 "0"
 ));
-// there would be 4 tasks, 2 for each task group
-
EasyMock.expect(taskClient.getCheckpointsAsync(EasyMock.contains("sequenceName-1"),
 EasyMock.anyBoolean()))
+// there would be 1 task, since there is only 1 shard
 
 Review comment:
   :+1:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8671: More Kinesis resharding adjustments

2019-10-15 Thread GitBox
clintropolis commented on a change in pull request #8671: More Kinesis 
resharding adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335218391
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1946,53 +1956,152 @@ private boolean updatePartitionDataFromStream()
   }
 }
 
-if (supportsPartitionExpiration()) {
-  // Look for expired shards and remove them from metadata storage and the 
partition groups
-  Set expiredPartitions = new HashSet<>();
-  for (PartitionIdType partitionTd : closedPartitions) {
-if (!partitionIds.contains(partitionTd)) {
-  expiredPartitions.add(partitionTd);
+if (!partitionIds.equals(previousPartitionIds)) {
+  // the set of partition IDs has changed, have any running tasks stop 
early so that we can adjust to the
+  // repartitioning quickly by creating new tasks
+  for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+if (!taskGroup.taskIds().isEmpty()) {
+  // a new partition was added and we are managing active tasks - set 
an early publish time 2 minutes in the
+  // future to give things time to settle
+  earlyStopTime = DateTimes.nowUtc().plusMinutes(2);
 
 Review comment:
   Where did this number come from, should it be a configuration, or is there 
any thing else like this in other parts of the codebase?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-15 Thread GitBox
clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335209789
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] KenjiTakahashi commented on issue #6468: Unexpected "Triggering JVM shutdown"

2019-10-15 Thread GitBox
KenjiTakahashi commented on issue #6468: Unexpected "Triggering JVM shutdown"
URL: 
https://github.com/apache/incubator-druid/issues/6468#issuecomment-542432036
 
 
   I am not working on this anymore, but as far as I remember, we just moved to 
using remote Tasks (instead of the local ones), so that they do not get killed 
when Overlord/Coordinator restarts.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] closed issue #6725: Ability to move leader to another node

2019-10-15 Thread GitBox
stale[bot] closed issue #6725: Ability to move leader to another node
URL: https://github.com/apache/incubator-druid/issues/6725
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] commented on issue #6725: Ability to move leader to another node

2019-10-15 Thread GitBox
stale[bot] commented on issue #6725: Ability to move leader to another node
URL: 
https://github.com/apache/incubator-druid/issues/6725#issuecomment-542422106
 
 
   This issue has been closed due to lack of activity. If you think that is 
incorrect, or the issue requires additional review, you can revive the issue at 
any time.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in InlineFirehose

2019-10-15 Thread GitBox
aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682#issuecomment-542422193
 
 
   @jihoonson thank you for your guidance.
   I will update the PR with the both of the points asap.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jihoonson commented on issue #8682: implement FiniteFirehoseFactory in InlineFirehose

2019-10-15 Thread GitBox
jihoonson commented on issue #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682#issuecomment-542421301
 
 
   @aditya-r-m thank you for your contribution! I have a couple of comments.
   
   - `InlineFirehoseFactory` should override `boolean isSplittable()` and 
return false because it's not splittable.
   - Would you please add some unit tests? Those test should verify 
`InlineFirehoseFactory` implements `FiniteFirehoseFactory` and `isSplittable` 
returns false. `InlineFirehoseFactory` is a good place for them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in InlineFirehose

2019-10-15 Thread GitBox
aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682#issuecomment-542421231
 
 
   @ccaominh @jihoonson the approach mentioned on the ticket is implemented 
here.
   I have tested it with basic inline ingestion tasks & will double check the 
behavior in a few more scenarios.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] aditya-r-m opened a new pull request #8682: implement FiniteFirehoseFactory in InlineFirehose

2019-10-15 Thread GitBox
aditya-r-m opened a new pull request #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682
 
 
   Fixes #8673.
   
   
   
   
   
   ### Description
   
   Implementation of FiniteFirehoseFactory to support index_parallel tasks with 
inline data ingestion.
   The implementation always returns 1 split.
   
   
   
   This PR has:
   - [x] been self-reviewed.
   
   
   
   # Key changed/added classes in this PR
* `InlineFirehoseFactory.java `
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on issue #8656: Message rejection absolute date

2019-10-15 Thread GitBox
himanshug commented on issue #8656: Message rejection absolute date
URL: https://github.com/apache/incubator-druid/pull/8656#issuecomment-542416405
 
 
   thanks, can you also update the necessary documentation and mention the 
precedence of `lateMessageRejectionStartDateTime` vs 
`lateMessageRejectionPeriod` or maybe there should be some check ensuring user 
doesn't specify both as that is most likely an user error.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] mohammadjkhan opened a new issue #8681: druid-basic-security.md is missing documentation for SSL configuration related properties

2019-10-15 Thread GitBox
mohammadjkhan opened a new issue #8681: druid-basic-security.md is missing 
documentation for SSL configuration related properties
URL: https://github.com/apache/incubator-druid/issues/8681
 
 
   druid-basic-security.md is missing documentation related to setting 
druid.auth.basic.ssl properties used for setting SSL related configuration 

   ### Affected Version
   Master (0.17.0)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

2019-10-15 Thread GitBox
himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335160549
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/common/task/BatchAppenderators.java
 ##
 @@ -59,13 +63,21 @@ public static Appenderator newAppenderator(
   TaskToolbox toolbox,
   DataSchema dataSchema,
   AppenderatorConfig appenderatorConfig,
+  FirehoseFactory firehoseFactory,
   DataSegmentPusher segmentPusher
   )
   {
+final boolean isReingest;
+if (firehoseFactory instanceof IngestSegmentFirehoseFactory) {
+  isReingest = 
dataSchema.getDataSource().equals(((IngestSegmentFirehoseFactory) 
firehoseFactory).getDataSource());
+} else {
+  isReingest = false;
+}
 
 Review comment:
   For now, I think this config is best left undocumented and for auto 
compaction internal usage only.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

2019-10-15 Thread GitBox
himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335159299
 
 

 ##
 File path: core/src/main/java/org/apache/druid/timeline/CompactionState.java
 ##
 @@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.timeline;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.indexer.partitions.PartitionsSpec;
+
+import java.util.Map;
+import java.util.Objects;
+
+public class CompactionState
+{
+  private final PartitionsSpec partitionsSpec;
+  // org.apache.druid.segment.IndexSpec cannot be used here to avoid the 
dependency cycle
 
 Review comment:
   got it, thanks.
   
   maybe in next round of module merge: merge core into processing if there is 
no use case of anyone depending on druid-core directly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

2019-10-15 Thread GitBox
himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335159299
 
 

 ##
 File path: core/src/main/java/org/apache/druid/timeline/CompactionState.java
 ##
 @@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.timeline;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.indexer.partitions.PartitionsSpec;
+
+import java.util.Map;
+import java.util.Objects;
+
+public class CompactionState
+{
+  private final PartitionsSpec partitionsSpec;
+  // org.apache.druid.segment.IndexSpec cannot be used here to avoid the 
dependency cycle
 
 Review comment:
   got it, thanks.
   
   maybe in next round of module merge: merge core into processing if there is 
no use case of anyone depending on druid-core directly :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

2019-10-15 Thread GitBox
himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335158395
 
 

 ##
 File path: core/src/main/java/org/apache/druid/timeline/CompactionState.java
 ##
 @@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.timeline;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.indexer.partitions.PartitionsSpec;
+
+import java.util.Map;
+import java.util.Objects;
+
+public class CompactionState
 
 Review comment:
   LGTM, thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-15 Thread GitBox
clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335153151
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
 
 Review comment:
   heh, thanks for reminding me to add some actually useful text here, I was 
also going to mention that functionally it is like a composite of 
`MergeSequence` and `CombiningSequence` :+1:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-15 Thread GitBox
himanshug commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335151691
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
 
 Review comment:
   not organic ? :-P
   
   maybe copy some part from 
https://github.com/apache/incubator-druid/issues/8577 also and add link to that 
.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-15 Thread GitBox
himanshug commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334687224
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   Not saying that we build the "alert" mechanism itself (e.g. sending mail 
etc) .. but use existing emitter based Alert mechanism.
   Also, router endpoint could just return the checkup results from previous 
run making web-console experience very fast .
   
   
   makes sense ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   [Update] Not saying that we build the "alert" mechanism itself (e.g. sending 
mail etc) .. but use existing emitter based Alert mechanism.
   Also, router endpoint could just return the checkup results from previous 
run making web-console experience very fast .
   
   
   makes sense ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   [Update] Not saying that we build the "alert" mechanism itself (e.g. sending 
mail etc) .. but use existing emitter based Alert mechanism.
   Also, router endpoint could just return the checkup results from previous 
run making web-console experience very fast .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] leventov opened a new issue #8680: Make IntelliJ's inspection "Method is identical to its super method" a error

2019-10-15 Thread GitBox
leventov opened a new issue #8680: Make IntelliJ's inspection "Method is 
identical to its super method" a error
URL: https://github.com/apache/incubator-druid/issues/8680
 
 
   28 occurrences in the codebase, currently.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   [Update] Not saying that we build the "alert" mechanism itself (e.g. sending 
mail etc) .. but use existing emitter based Alert mechanism.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
himanshug commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542365588
 
 
   @fjy are you reacting to something? Adjust if from what to what? Right now 
suggestions will show up in a neutral (dark) color and issues will show up in 
blueprint's warning style. You will get a red error if the tests early 
terminated due to an unrecoverable state (like in the screenshot I posted where 
the cluster is not even running).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky commented on issue #8674: Druid coordinator dynamic compaction failure

2019-10-15 Thread GitBox
vogievetsky commented on issue #8674: Druid coordinator dynamic compaction 
failure
URL: 
https://github.com/apache/incubator-druid/issues/8674#issuecomment-542363941
 
 
   What is you `druid.indexer.runner.type` set to?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly logging JsonMappingException when using druid-basic-security extension with an authenticator

2019-10-15 Thread GitBox
mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly 
logging JsonMappingException when using druid-basic-security extension with an 
authenticator that has no users setup
URL: 
https://github.com/apache/incubator-druid/issues/8678#issuecomment-542349688
 
 
   Hi @jon-wei and @nishantmonu51, Can one of you assign this ticket to me?
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vaibhav-imply opened a new issue #8679: Feature request to support GREATEST/LEAST function in Druid sql

2019-10-15 Thread GitBox
vaibhav-imply opened a new issue #8679: Feature request to support  
GREATEST/LEAST  function in Druid sql 
URL: https://github.com/apache/incubator-druid/issues/8679
 
 
   ### Description
   Currently GREATEST/LEAST post-aggregators are available via native druid 
query. It will be great to have GREATEST/LEAST implementation of 
post-aggregators as Druid SQL functions.
   
   ### Motivation
   
   It will make user's life easier as they always feel comfortable writing an 
SQL over JSON query.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] fjy commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
fjy commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542335390
 
 
   @vogievetsky can we adjust the color scheme to more accurately reflect the 
severity of an issue and minimize false scares? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542331870
 
 
   @himanshug you (and anyone) can always help by providing me with more 
trouble shooting tests to encode!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jihoonson commented on issue #8676: NumShards will cause broker can't load segments

2019-10-15 Thread GitBox
jihoonson commented on issue #8676: NumShards will cause broker can't load 
segments 
URL: 
https://github.com/apache/incubator-druid/issues/8676#issuecomment-542314018
 
 
   @licl2014 thank you for the report! I believe the same issue exists in 
master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly logging JsonMappingException when using druid-basic-security extension with an authenticator

2019-10-15 Thread GitBox
mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly 
logging JsonMappingException when using druid-basic-security extension with an 
authenticator that has no users setup
URL: 
https://github.com/apache/incubator-druid/issues/8678#issuecomment-542313054
 
 
   I already have a fix for this issue and will be creating a PR shortly


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid-website-src] fjy merged pull request #65: Clarify CLA policy.

2019-10-15 Thread GitBox
fjy merged pull request #65: Clarify CLA policy.
URL: https://github.com/apache/incubator-druid-website-src/pull/65
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] mohammadjkhan opened a new issue #8678: Non-coordinator services are repeatedly logging JsonMappingException when using druid-basic-security extension with an authenticator

2019-10-15 Thread GitBox
mohammadjkhan opened a new issue #8678: Non-coordinator services are repeatedly 
logging JsonMappingException when using druid-basic-security extension with an 
authenticator that has no users setup
URL: https://github.com/apache/incubator-druid/issues/8678
 
 
   ### Affected Version
   Detected in master (0.17.0), but also occurs in versions 0.12.0 and above.
   
   ### Description
   Issue occurs when using druid-basic-security extension and having an 
authenticator in the authenticator chain that has no users setup (not even the 
admin and druid_system internal users). We're seeing the following exception 
getting logged repeatedly within non-coordinator services:
   WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) 
in 901ms.
   com.fasterxml.jackson.databind.JsonMappingException: No content to map due 
to end-of-input
   
   - Cluster size
   Any
   
   - Configurations in use
   // Coordinator config
   -server
   -Xms256m
   -Xmx256m
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=/tmp/druid
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
   -XX:+UseG1GC
   -Dlog4j.configurationFile=/path/to/log4j2.debug.xml
   -Dorg.jboss.logging.provider=slf4j
   -Ddruid.service=druid/coordinator
   -Ddruid.coordinator.period=PT10S
   -Ddruid.coordinator.startDelay=PT5S
   -Ddruid.host=localhost
   -Ddruid.extensions.directory=/path/to/apache/druid/extensions/
   
-Ddruid.extensions.loadList=[\"mysql-metadata-storage\",\"druid-basic-security\"]
   -Ddruid.zk.service.host=localhost
   -Ddruid.zk.paths.base=/druid
   -Ddruid.metadata.storage.type=mysql
   
-Ddruid.metadata.storage.connector.connectURI="jdbc:mysql://localhost:3306/druid"
   -Ddruid.metadata.storage.connector.user=druid1
   -Ddruid.metadata.storage.connector.password=test
   -Ddruid.emitter=logging
   -Ddruid.emitter.logging.logLevel=debug
   -Ddruid.sql.enable=true
   -Ddruid.auth.authenticator.local.type=basic
   -Ddruid.auth.authenticator.local.skipOnFailure=true
   -Ddruid.auth.authenticator.local.initialAdminPassword=password1
   -Ddruid.auth.authenticator.local.initialInternalClientPassword=password2
   -Ddruid.auth.authenticator.local.credentialsValidator.type=metadata
   -Ddruid.extensions.directory=/path/to/apache/druid/extensions/
   
-Ddruid.extensions.loadList=[\"mysql-metadata-storage\",\"druid-basic-security\"]
   -Ddruid.auth.authenticatorChain=[\"local\",\"test\"]
   -Ddruid.auth.authenticator.local.type=basic
   -Ddruid.auth.authenticator.local.skipOnFailure=true
   -Ddruid.auth.authenticator.local.initialAdminPassword=password1
   -Ddruid.auth.authenticator.local.initialInternalClientPassword=password2
   -Ddruid.auth.authenticator.local.credentialsValidator.type=metadata
   -Ddruid.auth.authenticator.local.authorizerName=local
   -Ddruid.auth.authenticator.test.type=basic
   -Ddruid.auth.authenticator.test.skipOnFailure=true
   -Ddruid.auth.authenticator.test.credentialsValidator.type=metadata
   -Ddruid.auth.authenticator.test.authorizerName=test
   -Ddruid.auth.authorizers=[\"local\",\"test\"]
   -Ddruid.auth.authorizer.local.type=basic
   -Ddruid.auth.authorizer.local.roleProvider.type=metadata
   -Ddruid.auth.authorizer.test.type=basic
   -Ddruid.auth.authorizer.test.roleProvider.type=metadata
   -Ddruid.escalator.type=basic
   -Ddruid.escalator.internalClientUsername=druid_system
   -Ddruid.escalator.internalClientPassword=password2
   -Ddruid.escalator.authorizerName=local
   
   // Broker config
   -server
   -Xms1g
   -Xmx1g
   -XX:MaxDirectMemorySize=1792m
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=/tmp/druid
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
   -XX:+UseG1GC
   -Dlog4j.configurationFile=/path/to/log4j2.debug.xml
   -Dorg.jboss.logging.provider=slf4j
   -Ddruid.service=druid/broker
   -Ddruid.broker.http.numConnections=5
   -Ddruid.broker.cache.useCache=false
   -Ddruid.broker.cache.populateCache=false
   -Ddruid.server.http.numThreads=9
   -Ddruid.processing.buffer.sizeBytes=25600
   -Ddruid.processing.numThreads=2
   -Ddruid.cache.sizeInBytes=1000
   -Ddruid.host=localhost
   -Ddruid.extensions.directory=
   
-Ddruid.extensions.loadList=[\"mysql-metadata-storage\",\"druid-basic-security\"]
   -Ddruid.zk.service.host=localhost
   -Ddruid.zk.paths.base=/druid
   -Ddruid.metadata.storage.type=mysql
   
-Ddruid.metadata.storage.connector.connectURI="jdbc:mysql://localhost:3306/druid"
   -Ddruid.metadata.storage.connector.user=druid1
   -Ddruid.metadata.storage.connector.password=test
   -Ddruid.emitter=logging
   -Ddruid.emitter.logging.logLevel=debug
   -Ddruid.sql.enable=true
   -Ddruid.auth.authenticator.local.type=basic
   -Ddruid.auth.authenticator.local.skipOnFailure=true
   -Ddruid.auth.authenticator.local.initialAdminPassword=password1
   -Ddruid.auth.authenticator.local.initialInternalClientPassword=password2
   -Ddruid.auth.authenticator.local.credentialsValidator.type=metadata

[GitHub] [incubator-druid-website-src] gianm opened a new pull request #65: Clarify CLA policy.

2019-10-15 Thread GitBox
gianm opened a new pull request #65: Clarify CLA policy.
URL: https://github.com/apache/incubator-druid-website-src/pull/65
 
 
   Update the community & CLA pages to reflect Apache policy.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on issue #8672: Druid Doctor

2019-10-15 Thread GitBox
himanshug commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542291030
 
 
   I am somewhat illiterate when it comes to modern UI frameworks and 
practices, so couldn't review the code but :+1:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] teeram commented on issue #8663: Kafka indexing service duplicate entry exception in druid_pendingSegments

2019-10-15 Thread GitBox
teeram commented on issue #8663: Kafka indexing service duplicate entry 
exception in druid_pendingSegments
URL: 
https://github.com/apache/incubator-druid/issues/8663#issuecomment-542288337
 
 
   Hey @jihoonson, sorry for the delays in response. Hmm, I haven't turned on 
the minor compaction features mentioned in that issue, so I don't suspect that 
to be the culprit either.
   
   Unfortunately, I can't share those exact segment id's that are in conflict. 
But I am looking through those tables and can describe some of the oddities I 
am seeing. One thing I have noticed in the `druid_pendingSegments` table is 
that the id in conflict has a `created_date` that is typically from the 
previous day. For instance, I am getting an id conflict today (2019-10-15) for 
an entry in `druid_pendingSegments` with a `created_date` of 2019-10-14. That 
seems a little odd to me that the resumed Kafka indexing task would not attempt 
to place an entry in `druid_pendingSegments` with an updated value for the 
`created_date` field. There are other entries in the `druid_pendingSegments` 
table with a `created_date` field from today (2019-10-15).
   
   I did a quick query in the `druid_segments` table to see what segments were 
available for the time chunk in conflict. I found that all the segments in this 
table had been unpublished (`used = false`). This was the behavior I was 
expecting as I was dropping old segments before new ones were created.
   
   Thanks again for all your assistance!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] smokemonster99 commented on issue #4194: Kafka lag emitter - Kafka Indexing Service

2019-10-15 Thread GitBox
smokemonster99 commented on issue #4194: Kafka lag emitter - Kafka Indexing 
Service
URL: https://github.com/apache/incubator-druid/pull/4194#issuecomment-542287669
 
 
   @pjain1 thanks! That was it, I just needed to actually send something to 
kafka to get the metrics to show up in datadog via dogstatsd.
   
   Maybe you can help with this somewhat related question...My configured 
monitors such as jvm and sys work fine but I see there is a realtime monitor 
for ingestion, (org.apache.druid.segment.realtime.RealtimeMetricsMonitor) but 
when I configure it as a monitor on middlemanager, I get errors in the logs 
such as "No implementation for 
java.util.List was bound."
   
   Any ideas here? I am just looking for ingest metrics and thought that one 
was the best to use...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on issue #8673: InlineFirehose does not work with index_parallel ingestion

2019-10-15 Thread GitBox
gianm commented on issue #8673: InlineFirehose does not work with 
index_parallel ingestion
URL: 
https://github.com/apache/incubator-druid/issues/8673#issuecomment-542266767
 
 
   > If this gets fixed, is it possible to fix these cryptic messages that are 
sent out? I assume it's because of Jackson deserialization exceptions being 
regurgitated to the user?
   
   Yeah, those are Jackson exceptions, and it would be nice to make them more 
pleasant in a systematic way. I'm not sure if there's an easy way to do that 
without destroying the information in the error message or needing to parse 
their text. Maybe Jackson provides a way to get errors in a more programmatic 
way and we can write our own messages.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] sascha-coenen opened a new issue #8677: Misleading documentation/usage of "druid.indexing.doubleStorage" property

2019-10-15 Thread GitBox
sascha-coenen opened a new issue #8677: Misleading documentation/usage of 
"druid.indexing.doubleStorage" property
URL: https://github.com/apache/incubator-druid/issues/8677
 
 
   
   ### Affected Version
   
   0.11.x and following
   
   ### Description
   The release notes of Druid 0.11.0 
(https://github.com/apache/incubator-druid/releases/tag/druid-0.11.0) state 
that the "druid.indexing.doubleStorage" property could be specified in the 
common.runtime.properties file:
   "To enable Double column storage, set the following property in the common 
runtime properties"
   Furthermore, the Druid codebase contains example common.runtime.properties 
files in which this property is set:
   
https://github.com/apache/incubator-druid/blob/druid-0.16.0-incubating/examples/conf/druid/cluster/_common/common.runtime.properties
   
   However, both the Druid source code and also the Druid documentation seem to 
suggest that this property needs to be set as a java system property:
   in class ColumnHolder
   `
   static boolean storeDoubleAsFloat()
   {
 String value = System.getProperty(DOUBLE_STORAGE_TYPE_PROPERTY, "double");
 return !"double".equals(StringUtils.toLowerCase(value));
   }
   `
   
   Likewise the documentation correctly states:
   
https://github.com/apache/incubator-druid/blob/druid-0.16.0-incubating/docs/configuration/index.md
   
   "To keep the old format set the system-wide property 
druid.indexing.doubleStorage=float"
   
   
   In summary, this property only comes into effect if set as a java system 
property. Setting it within the commons.runtime.properties file has no effect.
   I propose to either remove the property from the common.runtime.properties 
example files and to correct the release notes OR to add support for specifying 
this property in the configuration files. I would prefer the later option.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] commented on issue #6790: chooseBestServer bug in CostBalancerStrategy

2019-10-15 Thread GitBox
stale[bot] commented on issue #6790: chooseBestServer bug in 
CostBalancerStrategy
URL: 
https://github.com/apache/incubator-druid/issues/6790#issuecomment-542224988
 
 
   This issue has been marked as stale due to 280 days of inactivity. It will 
be closed in 4 weeks if no further activity occurs. If this issue is still 
relevant, please simply write any comment. Even if closed, you can still revive 
the issue at any time or discuss it on the d...@druid.apache.org list. Thank 
you for your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] stale[bot] commented on issue #8098: Kafka lookup custom Json and Jq extractor

2019-10-15 Thread GitBox
stale[bot] commented on issue #8098: Kafka lookup custom Json and Jq extractor
URL: https://github.com/apache/incubator-druid/pull/8098#issuecomment-542224940
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 4 weeks if no further activity occurs. If you think that's 
incorrect or this pull request should instead be reviewed, please simply write 
any comment. Even if closed, you can still revive the PR at any time or discuss 
it on the d...@druid.apache.org list. Thank you for your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jnaous commented on issue #8673: InlineFirehose does not work with index_parallel ingestion

2019-10-15 Thread GitBox
jnaous commented on issue #8673: InlineFirehose does not work with 
index_parallel ingestion
URL: 
https://github.com/apache/incubator-druid/issues/8673#issuecomment-542217699
 
 
   If this gets fixed, is it possible to fix these cryptic messages that are
   sent out? I assume it's because of Jackson deserialization exceptions being
   regurgitated to the user?
   
   On Mon, Oct 14, 2019 at 8:53 PM Jihoon Son  wrote:
   
   > I was writing about the same comment with @ccaominh
   > . InlineFirehose could implement
   > FiniteFirehoseFactory but it should always run in the sequential mode.
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
   
   -- 
   Jad Naous
   Imply | VP R
   650-521-3425
   jad.na...@imply.io
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] licl2014 opened a new issue #8676: NumShards will cause broker can't load segments

2019-10-15 Thread GitBox
licl2014 opened a new issue #8676: NumShards will cause broker can't load 
segments 
URL: https://github.com/apache/incubator-druid/issues/8676
 
 
   NumShards will cause broker can't load segments 
   
   ### Affected Version
   
   0.9.2/0.12.3
   
   ### Description
   
   
   - We load batch data which set `NumShards` to 100
   - The number of batch data is 10,  and their cardinality is 10 too.
   - So although the reduce number is 100, but finally the shards will only 10 
,and their shards number is not continuous.
   - Broker will add them in `incompletePartitionsTimeline` when creating 
versionedIntervalTimeline.
   - The query about this datasource will return `null`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] leventov opened a new issue #8675: Document what does it mean for ServletFilterHolder.getDispatcherType() to return null

2019-10-15 Thread GitBox
leventov opened a new issue #8675: Document what does it mean for 
ServletFilterHolder.getDispatcherType() to return null
URL: https://github.com/apache/incubator-druid/issues/8675
 
 
   `ServletFilterHolder.getDispatcherType()` is annotated `@Nullable` but it 
doesn't explain in what cases it's OK (or not OK) to return null from this 
method for subclasses.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] mr0al opened a new issue #8674: Druid coordinator dynamic compaction failure

2019-10-15 Thread GitBox
mr0al opened a new issue #8674: Druid coordinator dynamic compaction failure
URL: https://github.com/apache/incubator-druid/issues/8674
 
 
   Druid 0.14.1
   
   I've set up druid dynamic compaction config on a realtime datasource but it 
seems this is even failing to launch a task. 
   If helps, I have long running indexer 5h with an intermediateHandoffPeriod 
of 5mins.
   Couldn't find anything anywhere on the below.. any ideas pls?
   
   ![Screen Shot 2019-10-15 at 10 06 49] 
(https://user-images.githubusercontent.com/5665595/66819286-a52d6980-ef36-11e9-92a3-1701b29dd083.png)
 
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] leventov commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-15 Thread GitBox
leventov commented on a change in pull request #8578: parallel broker merges on 
fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334845335
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] himanshug commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
himanshug commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334759043
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] weitianpei commented on issue #6468: Unexpected "Triggering JVM shutdown"

2019-10-14 Thread GitBox
weitianpei commented on issue #6468: Unexpected "Triggering JVM shutdown"
URL: 
https://github.com/apache/incubator-druid/issues/6468#issuecomment-542039470
 
 
   hi,KenjiTakahashi :
   I have met the same issue as you.Have you solved your problem?Can you help 
me.
   Please reply to me,my email is 342351...@qq.com   Thank you.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid-website-src] gianm opened a new pull request #64: Update Starter issues link.

2019-10-14 Thread GitBox
gianm opened a new pull request #64: Update Starter issues link.
URL: https://github.com/apache/incubator-druid-website-src/pull/64
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm commented on issue #713: Reported classloader issues with hdfs-storage-module

2019-10-14 Thread GitBox
gianm commented on issue #713: Reported classloader issues with 
hdfs-storage-module
URL: https://github.com/apache/incubator-druid/issues/713#issuecomment-542033779
 
 
   This isn't isn't relevant any longer. Hadoop jars aren't needed on Druid's 
main classpath.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] gianm closed issue #713: Reported classloader issues with hdfs-storage-module

2019-10-14 Thread GitBox
gianm closed issue #713: Reported classloader issues with hdfs-storage-module
URL: https://github.com/apache/incubator-druid/issues/713
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jihoonson commented on issue #8673: InlineFirehose does not work with index_parallel ingestion

2019-10-14 Thread GitBox
jihoonson commented on issue #8673: InlineFirehose does not work with 
index_parallel ingestion
URL: 
https://github.com/apache/incubator-druid/issues/8673#issuecomment-542024506
 
 
   I was writing about the same comment with @ccaominh. InlineFirehose could 
implement FiniteFirehoseFactory but it should always run in the sequential 
mode. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] ccaominh commented on issue #8673: InlineFirehose does not work with index_parallel ingestion

2019-10-14 Thread GitBox
ccaominh commented on issue #8673: InlineFirehose does not work with 
index_parallel ingestion
URL: 
https://github.com/apache/incubator-druid/issues/8673#issuecomment-542024078
 
 
   To work with parallel indexing, one possibility is to modify`InlineFirehose` 
to implement `FiniteFirehoseFactory` so that it is splittable and always 
returns 1 split (that contains the entire contents).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky commented on issue #8670: Issue with "Paste Data" data load

2019-10-14 Thread GitBox
vogievetsky commented on issue #8670: Issue with "Paste Data" data load
URL: 
https://github.com/apache/incubator-druid/issues/8670#issuecomment-542018032
 
 
   The error you are seeing is a bug and is a manifestation of 
https://github.com/apache/incubator-druid/issues/8673
   
   To get around it you simply need to turn off parallel indexing when using 
paste data (it would not add speed anyway)
   
   
![image](https://user-images.githubusercontent.com/177816/66797302-2b0dce00-eebf-11e9-997b-2b953ae8d48f.png)
   
   In the above step please make sure that the "Parallel indexing" switch is 
off (like in the screenshot).
   
   Also please note that the data sample you pasted in the issue would be 
sub-optimal for Druid, you would only really be able to parse it using the 
regexp parser and even then you would have issues with the first two rows. 
Unless you want to write your own parser it would be more advisable to use data 
in [line json](http://jsonlines.org/), CSV, or TSV formats.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky opened a new issue #8673: InlineFirehose does not work with index_parallel ingestion

2019-10-14 Thread GitBox
vogievetsky opened a new issue #8673: InlineFirehose does not work with 
index_parallel ingestion
URL: https://github.com/apache/incubator-druid/issues/8673
 
 
   As the title says, if you try to submit a task like:
   
   ```json
   {
 "type": "index_parallel",
 "spec": {
   "type": "index_parallel",
   "ioConfig": {
 "type": "index_parallel",
 "firehose": {
   "type": "inline",
   "data": "{\"name\":\"Vadim\"}"
 }
   },
   "tuningConfig": {
 "type": "index_parallel"
   },
   "dataSchema": {
 "dataSource": "some_data",
 "granularitySpec": {
   "type": "uniform",
   "queryGranularity": "HOUR",
   "rollup": true,
   "segmentGranularity": "DAY"
 },
 "parser": {
   "type": "string",
   "parseSpec": {
 "format": "json",
 "timestampSpec": {
   "column": "!!!_no_such_column_!!!",
   "missingValue": "2010-01-01T00:00:00Z"
 },
 "dimensionsSpec": {
   "dimensions": [
 "name"
   ]
 }
   }
 },
 "metricsSpec": [
   {
 "name": "count",
 "type": "count"
   }
 ]
   }
 }
   }
   ```
   
   You get an error of:
   
   ```
   {"error":"Instantiation of [simple type, class 
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask]
 value failed: [InlineFirehoseFactory] should implement FiniteFirehoseFactory"}
   ```
   
   However changing the ingestion to just `index`:
   
   ```json
   {
 "type": "index",
 "spec": {
   "type": "index",
   "ioConfig": {
 "type": "index",
 "firehose": {
   "type": "inline",
   "data": "{\"name\":\"Vadim\"}"
 }
   },
   "tuningConfig": {
 "type": "index"
   },
   "dataSchema": {
 "dataSource": "some_data",
 "granularitySpec": {
   "type": "uniform",
   "queryGranularity": "HOUR",
   "rollup": true,
   "segmentGranularity": "DAY"
 },
 "parser": {
   "type": "string",
   "parseSpec": {
 "format": "json",
 "timestampSpec": {
   "column": "!!!_no_such_column_!!!",
   "missingValue": "2010-01-01T00:00:00Z"
 },
 "dimensionsSpec": {
   "dimensions": [
 "name"
   ]
 }
   }
 },
 "metricsSpec": [
   {
 "name": "count",
 "type": "count"
   }
 ]
   }
 }
   }
   ```
   
   Makes it all work


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] vogievetsky opened a new pull request #8672: Druid Doctor

2019-10-14 Thread GitBox
vogievetsky opened a new pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672
 
 
   
![image](https://user-images.githubusercontent.com/177816/66796429-7a9eca80-eebc-11e9-8c45-f90a6d817c53.png)
   
   Adding a testing and troubleshooting framework to the console.
   
   The Druid Doctor is a dialog that runs a series of tests that test the 
capabilities of a cluster via the HTTP APIs available via the router node. The 
aim is to be able to identify common cluster and data misconfigurations and 
present helpful suggestions to the user.
   
   There are currently 9 tests:
   
   - Verify own status
   - Verify own runtime properties
   - Verify the Coordinator and Overlord status
   - Verify the Coordinator and Overlord runtime properties
   - Verify that the sampler works
   - Verify that SQL works
   - Verify that there are historical nodes
   - Verify that the historicals are not overfilled
   - Look for time chunks that could benefit from compaction
   
   In the future I would also like to add a test for task slots and pending 
tasks


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] himanshug commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
himanshug commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334687224
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] smildlzj commented on issue #6721: QueryableIndexes are not sorted

2019-10-14 Thread GitBox
smildlzj commented on issue #6721: QueryableIndexes are not sorted
URL: 
https://github.com/apache/incubator-druid/issues/6721#issuecomment-541998794
 
 
   > Hi @smildlzj, sorry for late response.
   > 
   > This error is just a sanity check and shouldn't happen in production. Do 
you still see the same error?
   
   Hi, it still happen


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] jon-wei opened a new pull request #8671: More Kinesis resharding adjustments

2019-10-14 Thread GitBox
jon-wei opened a new pull request #8671: More Kinesis resharding adjustments
URL: https://github.com/apache/incubator-druid/pull/8671
 
 
   This PR adjusts certain aspects of the Kinesis resharding fixes introduced 
in #8644:
   - #8644 changed the shardID -> partition group mapping to use hashing, to 
provide a fixed mapping. However, when shards can expire, this can result in 
imbalanced distribution of active shards among tasks (e.g., some tasks may not 
get any readable shards). This PR reverts the mapping logic back to using the 
position of the shard within the shard list % task count. To handle expired 
shards, this PR instead recomputes the partition group mappings when shards 
expire, and tells any active tasks created under the old mappings to exit early.
   - Instead of deleting expired shards from metadata, this PR changes the 
expiration handling to store an "EXPIRED" sequence number marker instead. While 
expired shards should never reappear, keeping a record of expired shards in 
metadata will us to better handle such anomalies.
   - Adjusts `checkSequenceNumberAvailability` (renamed to 
`checkOffsetAvailability`), only the earliest available offset needs to be 
compared against the target offset. Because the "LATEST" Kinesis shard iterator 
only provides access to the seq no after the most recent record (and there is 
no mode to get the latest readable record), the "get latest offset" operation 
in Kinesis runs in a polling loop. When data is not being continuously written 
to a shard, this can result in a Kinesis supervisor making extremely slow 
progress as it waits for the polling loop to timeout for each shard that has no 
recent data. Removing the latest offset check restricts this problem situation 
to only when a Kinesis supervisor first discovers a shard that's empty (after 
data is written to that shard once, the earliest seq no check can return 
immediately)
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added documentation for new or modified features or behaviors.
   - [x] added Javadocs for most classes and all non-trivial methods. Linked 
related entities via Javadoc links.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths.
   - [x] been tested in a test Druid cluster.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334718832
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1077 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
jihoonson commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334717418
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1077 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334715834
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
jihoonson commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334714150
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1077 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
jihoonson commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334712882
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1077 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
jihoonson commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334712933
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1077 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8578: parallel broker merges on fork join pool

2019-10-14 Thread GitBox
jihoonson commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334711623
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1077 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] jihoonson commented on issue #8573: Stateful auto compaction

2019-10-14 Thread GitBox
jihoonson commented on issue #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#issuecomment-541965075
 
 
   Some docs should be further updated, but I will do it once this PR and 
https://github.com/apache/incubator-druid/pull/8570 are merged.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org



  1   2   3   4   5   6   7   8   9   10   >