[GitHub] [incubator-druid] jihoonson closed issue #8489: Stateful auto compaction

[GitBox]

jihoonson closed issue #8489: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/issues/8489
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] jihoonson merged pull request #8573: Stateful auto compaction

[GitBox]

jihoonson merged pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid] branch master updated (1a78a0c -> 4046c86)

[jihoonson]

This is an automated email from the ASF dual-hosted git repository.

jihoonson pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git.


from 1a78a0c  Add credentials for ECS (#8651)
 add 4046c86  Stateful auto compaction (#8573)

No new revisions were added by this update.

Summary of changes:
 .../query/CachingClusteredClientBenchmark.java |   3 +-
 .../NewestSegmentFirstPolicyBenchmark.java |   8 +-
 .../indexer/partitions/DynamicPartitionsSpec.java  |  22 ++-
 .../org/apache/druid/timeline/CompactionState.java |  97 +
 .../org/apache/druid/timeline/DataSegment.java | 110 ++-
 ...LoadSpec.java => PruneLastCompactionState.java} |  12 +-
 .../org/apache/druid/timeline/DataSegmentTest.java |  43 ++
 .../SegmentWithOvershadowedStatusTest.java |   8 +-
 docs/configuration/index.md|   3 +-
 docs/design/coordinator.md |  49 ---
 docs/ingestion/data-management.md  |   2 -
 .../storage/hdfs/HdfsDataSegmentPusherTest.java|   3 +-
 .../kafka/KafkaIndexTaskTuningConfigTest.java  |  27 +---
 .../druid/indexer/BatchDeltaIngestionTest.java |   3 +-
 ...onSpecUpdateDatasourcePathSpecSegmentsTest.java |   3 +-
 .../index/RealtimeAppenderatorTuningConfig.java|   1 +
 .../task/AppenderatorDriverRealtimeIndexTask.java  |   3 +-
 .../indexing/common/task/BatchAppenderators.java   |  10 +-
 .../druid/indexing/common/task/CompactionTask.java | 141 +++
 .../common/task/FiniteFirehoseProcessor.java   |   2 +-
 .../druid/indexing/common/task/IndexTask.java  |   4 +-
 .../apache/druid/indexing/common/task/Tasks.java   |   5 +
 .../batch/parallel/PartialSegmentGenerateTask.java |   3 +-
 .../task/batch/parallel/SinglePhaseSubTask.java|   5 +-
 .../SeekableStreamIndexTaskRunner.java |   3 +-
 .../SeekableStreamIndexTaskTuningConfig.java   |   1 +
 .../apache/druid/indexing/common/TestUtils.java|   4 +-
 .../common/task/ClientCompactQuerySerdeTest.java   |   2 -
 .../common/task/CompactionTaskRunTest.java |  23 
 .../indexing/common/task/CompactionTaskTest.java   |  93 ++---
 .../common/task/TestAppenderatorsManager.java  |   2 +
 .../results/auth_test_sys_schema_segments.json |   2 +-
 .../java/org/apache/druid/segment/TestHelper.java  |   4 +-
 .../druid/client/indexing/ClientCompactQuery.java  |  16 +--
 .../indexing/ClientCompactQueryTuningConfig.java   |   5 +
 .../client/indexing/HttpIndexingServiceClient.java |   4 +-
 .../client/indexing/IndexingServiceClient.java |   1 -
 .../segment/indexing/RealtimeTuningConfig.java |   7 +
 .../druid/segment/realtime/FireDepartment.java |   6 -
 .../realtime/appenderator/AppenderatorConfig.java  |   3 +
 .../realtime/appenderator/AppenderatorImpl.java|  66 ++---
 .../realtime/appenderator/Appenderators.java   |  29 ++--
 .../appenderator/AppenderatorsManager.java |   1 +
 .../DefaultOfflineAppenderatorFactory.java |  11 +-
 .../DummyForInjectionAppenderatorsManager.java |   1 +
 .../appenderator/PeonAppenderatorsManager.java |   2 +
 .../UnifiedIndexerAppenderatorsManager.java|  30 +++--
 .../segment/realtime/plumber/RealtimePlumber.java  |   1 +
 .../druid/segment/realtime/plumber/Sink.java   |  53 ++--
 .../coordinator/DataSourceCompactionConfig.java|  62 -
 .../druid/server/coordinator/DruidCoordinator.java |  11 +-
 .../helper/DruidCoordinatorSegmentCompactor.java   |  32 +++--
 .../helper/NewestSegmentFirstIterator.java | 150 +++--
 .../helper/NewestSegmentFirstPolicy.java   |  10 +-
 .../coordinator/helper/SegmentCompactorUtil.java   |  25 
 .../druid/client/CachingClusteredClientTest.java   |   1 +
 .../druid/client/ImmutableDruidDataSourceTest.java |  25 ++--
 .../client/indexing/NoopIndexingServiceClient.java |   1 -
 .../overlord/SegmentPublishResultTest.java |   4 +-
 .../org/apache/druid/server/ServerTestHelper.java  |   4 +-
 .../coordination/SegmentChangeRequestDropTest.java |   2 +-
 .../coordination/SegmentChangeRequestLoadTest.java |   2 +-
 .../BatchDataSegmentAnnouncerTest.java |   2 +-
 .../coordinator/CuratorDruidCoordinatorTest.java   |   6 +-
 .../DataSourceCompactionConfigTest.java|  53 
 .../server/coordinator/DruidCoordinatorTest.java   |   3 +-
 .../DruidCoordinatorSegmentCompactorTest.java  |  27 +++-
 .../helper/NewestSegmentFirstPolicyTest.java   |  73 +-
 .../druid/server/http/ServersResourceTest.java |   4 +-
 .../main/java/org/apache/druid/cli/CliBroker.java  |   2 +
 .../java/org/apache/druid/cli/CliCoordinator.java  |   3 +-
 .../java/org/apache/druid/cli/CliHistorical.java   |   2 +
 .../org/apache/druid/cli/CliMiddleManager.java |   2 +
 .../java/org/apache/druid/cli/ExportMetadata.java  |   3 +-
 

[GitHub] [incubator-druid] gianm commented on issue #8672: Druid Doctor

[GitBox]

gianm commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542490718
 
 
   > Can I suggest using something like 
https://metrics.dropwizard.io/3.1.0/manual/healthchecks/?
   
   It looks like something small but useful. We recently started using 
dropwizard in a couple other places and it has been nice.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

[GitBox]

vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542489399
 
 
   Great feedback! Thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid-website] vogievetsky merged pull request #40: Autobuild

[GitBox]

vogievetsky merged pull request #40: Autobuild
URL: https://github.com/apache/incubator-druid-website/pull/40
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid-website] branch asf-site updated (610126d -> 830aac8)

[vogievetsky]

This is an automated email from the ASF dual-hosted git repository.

vogievetsky pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/incubator-druid-website.git.


from 610126d  Autobuild (#39)
 new 432ea60  Autobuild
 new 65c0854  add missing files
 new 830aac8  Merge pull request #40 from implydata/autobuild

The 107 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 community/cla.html   | 16 
 community/index.html |  6 +-
 index.html   | 17 -
 3 files changed, 5 insertions(+), 34 deletions(-)


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid-website] vogievetsky commented on issue #40: Autobuild

[GitBox]

vogievetsky commented on issue #40: Autobuild
URL: 
https://github.com/apache/incubator-druid-website/pull/40#issuecomment-542489172
 
 
   Self merging build website update.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid-website] vogievetsky opened a new pull request #40: Autobuild

[GitBox]

vogievetsky opened a new pull request #40: Autobuild
URL: https://github.com/apache/incubator-druid-website/pull/40
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254646
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
 
 Review comment:
   "from the Router"?
   
   Also, "something must be broken" isn't adding much. If we have any 
speculation about why this might happen, add it here, otherwise I think we 
don't have much to say.
   
   Btw, I'd suggest checking for an unauthorized response and translate that 
into something like "You are not authorized to access the XXX API." (for every 
case where the response is generated). This way it'll degrade nicely for secure 
clusters.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335255062
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
+);
+  }
+
+  // Check that the underlying Java is Java 8 the only officially 
supported Java version at the moment.
+  if (
+properties['java.runtime.version'] &&
+!properties['java.runtime.version'].startsWith('1.8')
+  ) {
+controls.addSuggestion(
+  `It looks like are running Java 
${properties['java.runtime.version']}, Druid only officially supports Java 
1.8.x`,
 
 Review comment:
   `java.specification.version` may be easier to parse. (startsWith 1.8 is a 
little brittle, what happens when 1.80 is out )
   
   A period seems more grammatical to me here (vs. a comma).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335256239
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
+);
+  }
+
+  // Check that the underlying Java is Java 8 the only officially 
supported Java version at the moment.
+  if (
+properties['java.runtime.version'] &&
+!properties['java.runtime.version'].startsWith('1.8')
+  ) {
+controls.addSuggestion(
+  `It looks like are running Java 
${properties['java.runtime.version']}, Druid only officially supports Java 
1.8.x`,
+);
+  }
+
+  // Check that "user.timezone"
+  if (properties['user.timezone'] && properties['user.timezone'] !== 
'UTC') {
+controls.addSuggestion(
+  `It looks like "user.timezone" is set to 
${properties['user.timezone']}, it is recommended to set this to "UTC"`,
+);
+  }
+},
+  },
+
+  // -
+  // Coordinator and Overlord
+  // -
+  {
+name: 'Verify the Coordinator and Overlord status',
+check: async controls => {
+  // Make sure that everything in Coordinator's /status is good
+  let myStatus: any;
+  try {
+myStatus = (await axios.get(`/status`)).data;
+  } catch {
+return;
+  }
+
+  let coordinatorStatus: any;
+  try {
+coordinatorStatus = (await 
axios.get(`/proxy/coordinator/status`)).data;
+

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335252556
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
+
+  if (!this.mounted) return;
+  try {
+await check.check({
+  addSuggestion: (message: string) => {
+addToDiagnoses({
+  type: 'suggestion',
+  check: check.name,
+  message,
+});
+  },
+  addIssue: (message: string) => {
+addToDiagnoses({
+  type: 'issue',
+  check: check.name,
+  message,
+});
+  },
+  terminateChecks: () => {
+if (!this.mounted) return;
+this.setState({
+  earlyTermination: `${check.name} early terminated the check 
suite`,
+});
+terminateChecks = true;
+  },
+});
+  } catch (e) {
+addToDiagnoses({
+  type: 'issue',
+  check: check.name,
+  message: `${check.name} encountered an unhandled exception`,
 
 Review comment:
   What might cause this? (Anyone that sees it will be left scratching their 
heads, maybe we can help them out a bit)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254173
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
 
 Review comment:
   How about using "Router" not "cluster" here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335253407
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
+
+  if (!this.mounted) return;
+  try {
+await check.check({
 
 Review comment:
   Does anything get printed for checks that find no issues? Should it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335252958
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
 
 Review comment:
   Is this so they appear slowly and look like they are really working hard?
   
   If so, lol.
   
   Also, I dunno, maybe don't do it. Assuming it's milliseconds, 500ms is a 
long time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335252410
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-dialog.tsx
 ##
 @@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { Button, Callout, Classes, Dialog, Intent } from '@blueprintjs/core';
+import { IconNames } from '@blueprintjs/icons';
+import React from 'react';
+
+import { delay, pluralIfNeeded } from '../../utils';
+
+import { DOCTOR_CHECKS } from './doctor-checks';
+
+import './doctor-dialog.scss';
+
+interface Diagnosis {
+  type: 'suggestion' | 'issue';
+  check: string;
+  message: string;
+}
+
+export interface DoctorDialogProps {
+  onClose: () => void;
+}
+
+export interface DoctorDialogState {
+  currentCheckIndex?: number;
+  diagnoses?: Diagnosis[];
+  earlyTermination?: string;
+}
+
+export class DoctorDialog extends React.PureComponent {
+  private mounted = false;
+
+  constructor(props: DoctorDialogProps, context: any) {
+super(props, context);
+this.state = {};
+  }
+
+  componentDidMount(): void {
+this.mounted = true;
+  }
+
+  componentWillUnmount(): void {
+this.mounted = false;
+  }
+
+  async doChecks() {
+this.setState({ currentCheckIndex: 0, diagnoses: [] });
+
+const addToDiagnoses = (diagnosis: Diagnosis) => {
+  if (!this.mounted) return;
+  this.setState(oldState => ({
+diagnoses: (oldState.diagnoses || []).concat(diagnosis),
+  }));
+};
+
+for (let i = 0; i < DOCTOR_CHECKS.length; i++) {
+  if (!this.mounted) return;
+  this.setState({ currentCheckIndex: i });
+  const check = DOCTOR_CHECKS[i];
+  let terminateChecks = false;
+
+  // Slow down a bit so that the user can read the test name
+  await delay(500);
+
+  if (!this.mounted) return;
+  try {
+await check.check({
+  addSuggestion: (message: string) => {
+addToDiagnoses({
+  type: 'suggestion',
+  check: check.name,
+  message,
+});
+  },
+  addIssue: (message: string) => {
+addToDiagnoses({
+  type: 'issue',
+  check: check.name,
+  message,
+});
+  },
+  terminateChecks: () => {
+if (!this.mounted) return;
+this.setState({
+  earlyTermination: `${check.name} early terminated the check 
suite`,
 
 Review comment:
   This line made the screenshot confusing to me (I had a moment like, what 
does "Verify own status early terminated" mean? We want to verify that the 
status has early terminated?).
   
   Maybe be more explicit that the check name is part of the message. Also, 
what does it mean when a check early terminated the check suite? Is it because 
the check found a problem? Or because it couldn't run at all? (This should be 
obvious through the message)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335253927
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
 
 Review comment:
   This could be too aggressive. It's fine if different servers have different 
java versions, and would be normal if you're rolling a java upgrade through 
your servers, or if you just simply aren't fastidious about keeping them all at 
exactly the same version. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335255846
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
+);
+  }
+
+  // Check that the underlying Java is Java 8 the only officially 
supported Java version at the moment.
+  if (
+properties['java.runtime.version'] &&
+!properties['java.runtime.version'].startsWith('1.8')
+  ) {
+controls.addSuggestion(
+  `It looks like are running Java 
${properties['java.runtime.version']}, Druid only officially supports Java 
1.8.x`,
+);
+  }
+
+  // Check that "user.timezone"
+  if (properties['user.timezone'] && properties['user.timezone'] !== 
'UTC') {
+controls.addSuggestion(
+  `It looks like "user.timezone" is set to 
${properties['user.timezone']}, it is recommended to set this to "UTC"`,
+);
+  }
+},
+  },
+
+  // -
+  // Coordinator and Overlord
+  // -
+  {
+name: 'Verify the Coordinator and Overlord status',
+check: async controls => {
+  // Make sure that everything in Coordinator's /status is good
+  let myStatus: any;
+  try {
+myStatus = (await axios.get(`/status`)).data;
+  } catch {
+return;
+  }
+
+  let coordinatorStatus: any;
+  try {
+coordinatorStatus = (await 
axios.get(`/proxy/coordinator/status`)).data;
+

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254221
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
 
 Review comment:
   "from the Router"?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on a change in pull request #8672: Druid Doctor

[GitBox]

gianm commented on a change in pull request #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#discussion_r335254944
 
 

 ##
 File path: web-console/src/dialogs/doctor-dialog/doctor-checks.tsx
 ##
 @@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import axios from 'axios';
+
+import { pluralIfNeeded, queryDruidSql } from '../../utils';
+import { deepGet } from '../../utils/object-change';
+import { postToSampler } from '../../utils/sampler';
+
+export interface CheckControls {
+  addSuggestion: (message: string) => void;
+  addIssue: (message: string) => void;
+  terminateChecks: () => void;
+}
+
+export interface DoctorCheck {
+  name: string;
+  check: (controls: CheckControls) => Promise;
+}
+
+const RUNTIME_PROPERTIES_ALL_NODES_MUST_AGREE_ON: string[] = [
+  'user.timezone',
+  'druid.zk.service.host',
+];
+
+const RUNTIME_PROPERTIES_ALL_NODES_SHOULD_AGREE_ON: string[] = 
['java.version'];
+
+// In the future (when we can query other nodes) is will also be cool to check:
+// 'druid.storage.type' <=> historicals, overlords, mm
+// 'druid.indexer.logs.type' <=> overlord, mm, + peons
+
+const RUNTIME_PROPERTIES_MASTER_NODES_SHOULD_AGREE_ON: string[] = [
+  'druid.metadata.storage.type', // overlord + coordinator
+  'druid.metadata.storage.connector.connectURI',
+];
+
+export const DOCTOR_CHECKS: DoctorCheck[] = [
+  // -
+  // Self (router) checks
+  // -
+  {
+name: 'Verify own status',
+check: async controls => {
+  // Make sure that the router responds to /status and gives some valid 
info back
+  let status: any;
+  try {
+status = (await axios.get(`/status`)).data;
+  } catch (e) {
+controls.addIssue(
+  `Did not get a /status response, is the cluster running? Got: 
${e.message}`,
+);
+controls.terminateChecks();
+return;
+  }
+
+  if (typeof status.version !== 'string') {
+controls.addIssue('Could not get a valid /status response.');
+  }
+},
+  },
+  {
+name: 'Verify own runtime properties',
+check: async controls => {
+  // Make sure that everything in /status/properties is above board
+  let properties: Record;
+  try {
+properties = (await axios.get(`/status/properties`)).data;
+  } catch (e) {
+controls.addIssue('Did not get a /status/properties response, 
something must be broken.');
+return;
+  }
+
+  // Check that the management proxy is on, it really should be for 
someone to access the console in the first place but everything could happen
+  if (properties['druid.router.managementProxy.enabled'] !== 'true') {
+controls.addIssue(
+  `The router's "druid.router.managementProxy.enabled" is not reported 
as "true" that is unusual.`,
 
 Review comment:
   I would go with different language, like telling the user that we recommend 
setting this property in order for Coordinator / Overlord APIs to be accessible 
through the Router (and therefore the web console).
   
   The general idea is we should give people some hints about why each check is 
there and what they should do about them if issues are found.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8671: More Kinesis resharding adjustments

[GitBox]

jon-wei commented on a change in pull request #8671: More Kinesis resharding 
adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335254796
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1946,53 +1956,152 @@ private boolean updatePartitionDataFromStream()
   }
 }
 
-if (supportsPartitionExpiration()) {
-  // Look for expired shards and remove them from metadata storage and the 
partition groups
-  Set expiredPartitions = new HashSet<>();
-  for (PartitionIdType partitionTd : closedPartitions) {
-if (!partitionIds.contains(partitionTd)) {
-  expiredPartitions.add(partitionTd);
+if (!partitionIds.equals(previousPartitionIds)) {
+  // the set of partition IDs has changed, have any running tasks stop 
early so that we can adjust to the
+  // repartitioning quickly by creating new tasks
+  for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+if (!taskGroup.taskIds().isEmpty()) {
+  // a new partition was added and we are managing active tasks - set 
an early publish time 2 minutes in the
+  // future to give things time to settle
+  earlyStopTime = DateTimes.nowUtc().plusMinutes(2);
 
 Review comment:
   It makes sense to support that for Kafka as well (partitions can be added), 
but I'll do that in a follow-on PR


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] jon-wei commented on a change in pull request #8671: More Kinesis resharding adjustments

[GitBox]

jon-wei commented on a change in pull request #8671: More Kinesis resharding 
adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335254721
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1946,53 +1956,152 @@ private boolean updatePartitionDataFromStream()
   }
 }
 
-if (supportsPartitionExpiration()) {
-  // Look for expired shards and remove them from metadata storage and the 
partition groups
-  Set expiredPartitions = new HashSet<>();
-  for (PartitionIdType partitionTd : closedPartitions) {
-if (!partitionIds.contains(partitionTd)) {
-  expiredPartitions.add(partitionTd);
+if (!partitionIds.equals(previousPartitionIds)) {
+  // the set of partition IDs has changed, have any running tasks stop 
early so that we can adjust to the
+  // repartitioning quickly by creating new tasks
+  for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+if (!taskGroup.taskIds().isEmpty()) {
+  // a new partition was added and we are managing active tasks - set 
an early publish time 2 minutes in the
+  // future to give things time to settle
+  earlyStopTime = DateTimes.nowUtc().plusMinutes(2);
 
 Review comment:
   I updated this to use a configurable `repartitionTransitionDuration` 
property under the supervisor tuning config, along with docs on what the 
property is for (giving the stream time to write records to the new shards 
before transitioning to a new set of tasks)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] jnaous commented on issue #8672: Druid Doctor

[GitBox]

jnaous commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542479729
 
 
   Can I suggest using something like
   https://metrics.dropwizard.io/3.1.0/manual/healthchecks/?
   
   On Tue, Oct 15, 2019 at 6:47 PM Vadim Ogievetsky 
   wrote:
   
   > @himanshug  I 100% agree and I know @gianm
   >  will agree with you also. In fact @gianm
   >  was advocating for the checks should live in
   > Java and I was like: "do I look like James Gosling to you?"
   >
   > But in all seriousness I think you should look at this PR as a working
   > prototype. I want to verify/prove that a 1 button troubleshooting wizard
   > would be a useful addition to the community. I also wanted to have a
   > platform to go around the Druid devs and support ppl that I know to solicit
   > check from them.
   >
   > I think if this proves to be useful then the next step would be to move
   > the majority of the tests into Java and expose them as an endpoint. I would
   > make an issue to track that as soon as this PR is merged.
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
   
   -- 
   Jad Naous
   Imply | VP R
   650-521-3425
   jad.na...@imply.io
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

[GitBox]

vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542474187
 
 
   @himanshug I 100% agree and I know @gianm will agree with you also. In fact 
@gianm was advocating for the checks should live in Java and I was like: "do I 
look like James Gosling to you?"
   
   But in all seriousness I think you should look at this PR as a working 
prototype. I want to verify/prove that a 1 button troubleshooting wizard would 
be a useful addition to the community. I also wanted to have a platform to go 
around the Druid devs and support ppl that I know to solicit check from them.
   
   I think if this proves to be useful then the next step would be to move the 
majority of the tests into Java and expose them as an endpoint. I would make an 
issue to track that as soon as this PR is merged.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] SandishKumarHN commented on issue #8558: 7227 : Prohibit Non Final Static Field

[GitBox]

SandishKumarHN commented on issue #8558: 7227 : Prohibit Non Final Static Field
URL: https://github.com/apache/incubator-druid/pull/8558#issuecomment-542450809
 
 
   @leventov inspection failure from 
   `
   SeekableStreamSupervisor.java (4)
   175: TaskGroup() Optional used as type for parameter 
'minimumMessageTime'
   `
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8671: More Kinesis resharding adjustments

[GitBox]

clintropolis commented on a change in pull request #8671: More Kinesis 
resharding adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335219624
 
 

 ##
 File path: 
extensions-core/kinesis-indexing-service/src/test/java/org/apache/druid/indexing/kinesis/supervisor/KinesisSupervisorTest.java
 ##
 @@ -3885,8 +3906,8 @@ public void testShardSplit() throws Exception
 SHARD_ID0,
 "0"
 ));
-// there would be 4 tasks, 2 for each task group
-
EasyMock.expect(taskClient.getCheckpointsAsync(EasyMock.contains("sequenceName-1"),
 EasyMock.anyBoolean()))
+// there would be 1 task, since there is only 1 shard
 
 Review comment:
   :+1:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8671: More Kinesis resharding adjustments

[GitBox]

clintropolis commented on a change in pull request #8671: More Kinesis 
resharding adjustments
URL: https://github.com/apache/incubator-druid/pull/8671#discussion_r335218391
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java
 ##
 @@ -1946,53 +1956,152 @@ private boolean updatePartitionDataFromStream()
   }
 }
 
-if (supportsPartitionExpiration()) {
-  // Look for expired shards and remove them from metadata storage and the 
partition groups
-  Set expiredPartitions = new HashSet<>();
-  for (PartitionIdType partitionTd : closedPartitions) {
-if (!partitionIds.contains(partitionTd)) {
-  expiredPartitions.add(partitionTd);
+if (!partitionIds.equals(previousPartitionIds)) {
+  // the set of partition IDs has changed, have any running tasks stop 
early so that we can adjust to the
+  // repartitioning quickly by creating new tasks
+  for (TaskGroup taskGroup : activelyReadingTaskGroups.values()) {
+if (!taskGroup.taskIds().isEmpty()) {
+  // a new partition was added and we are managing active tasks - set 
an early publish time 2 minutes in the
+  // future to give things time to settle
+  earlyStopTime = DateTimes.nowUtc().plusMinutes(2);
 
 Review comment:
   Where did this number come from, should it be a configuration, or is there 
any thing else like this in other parts of the codebase?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

[GitBox]

clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335209789
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] KenjiTakahashi commented on issue #6468: Unexpected "Triggering JVM shutdown"

[GitBox]

KenjiTakahashi commented on issue #6468: Unexpected "Triggering JVM shutdown"
URL: 
https://github.com/apache/incubator-druid/issues/6468#issuecomment-542432036
 
 
   I am not working on this anymore, but as far as I remember, we just moved to 
using remote Tasks (instead of the local ones), so that they do not get killed 
when Overlord/Coordinator restarts.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] stale[bot] closed issue #6725: Ability to move leader to another node

[GitBox]

stale[bot] closed issue #6725: Ability to move leader to another node
URL: https://github.com/apache/incubator-druid/issues/6725
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] stale[bot] commented on issue #6725: Ability to move leader to another node

[GitBox]

stale[bot] commented on issue #6725: Ability to move leader to another node
URL: 
https://github.com/apache/incubator-druid/issues/6725#issuecomment-542422106
 
 
   This issue has been closed due to lack of activity. If you think that is 
incorrect, or the issue requires additional review, you can revive the issue at 
any time.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in InlineFirehose

[GitBox]

aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682#issuecomment-542422193
 
 
   @jihoonson thank you for your guidance.
   I will update the PR with the both of the points asap.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] jihoonson commented on issue #8682: implement FiniteFirehoseFactory in InlineFirehose

[GitBox]

jihoonson commented on issue #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682#issuecomment-542421301
 
 
   @aditya-r-m thank you for your contribution! I have a couple of comments.
   
   - `InlineFirehoseFactory` should override `boolean isSplittable()` and 
return false because it's not splittable.
   - Would you please add some unit tests? Those test should verify 
`InlineFirehoseFactory` implements `FiniteFirehoseFactory` and `isSplittable` 
returns false. `InlineFirehoseFactory` is a good place for them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in InlineFirehose

[GitBox]

aditya-r-m commented on issue #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682#issuecomment-542421231
 
 
   @ccaominh @jihoonson the approach mentioned on the ticket is implemented 
here.
   I have tested it with basic inline ingestion tasks & will double check the 
behavior in a few more scenarios.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] aditya-r-m opened a new pull request #8682: implement FiniteFirehoseFactory in InlineFirehose

[GitBox]

aditya-r-m opened a new pull request #8682: implement FiniteFirehoseFactory in 
InlineFirehose
URL: https://github.com/apache/incubator-druid/pull/8682
 
 
   Fixes #8673.
   
   
   
   
   
   ### Description
   
   Implementation of FiniteFirehoseFactory to support index_parallel tasks with 
inline data ingestion.
   The implementation always returns 1 split.
   
   
   
   This PR has:
   - [x] been self-reviewed.
   
   
   
   # Key changed/added classes in this PR
* `InlineFirehoseFactory.java `
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on issue #8656: Message rejection absolute date

[GitBox]

himanshug commented on issue #8656: Message rejection absolute date
URL: https://github.com/apache/incubator-druid/pull/8656#issuecomment-542416405
 
 
   thanks, can you also update the necessary documentation and mention the 
precedence of `lateMessageRejectionStartDateTime` vs 
`lateMessageRejectionPeriod` or maybe there should be some check ensuring user 
doesn't specify both as that is most likely an user error.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] mohammadjkhan opened a new issue #8681: druid-basic-security.md is missing documentation for SSL configuration related properties

[GitBox]

mohammadjkhan opened a new issue #8681: druid-basic-security.md is missing 
documentation for SSL configuration related properties
URL: https://github.com/apache/incubator-druid/issues/8681
 
 
   druid-basic-security.md is missing documentation related to setting 
druid.auth.basic.ssl properties used for setting SSL related configuration 

   ### Affected Version
   Master (0.17.0)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

[GitBox]

himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335160549
 
 

 ##
 File path: 
indexing-service/src/main/java/org/apache/druid/indexing/common/task/BatchAppenderators.java
 ##
 @@ -59,13 +63,21 @@ public static Appenderator newAppenderator(
   TaskToolbox toolbox,
   DataSchema dataSchema,
   AppenderatorConfig appenderatorConfig,
+  FirehoseFactory firehoseFactory,
   DataSegmentPusher segmentPusher
   )
   {
+final boolean isReingest;
+if (firehoseFactory instanceof IngestSegmentFirehoseFactory) {
+  isReingest = 
dataSchema.getDataSource().equals(((IngestSegmentFirehoseFactory) 
firehoseFactory).getDataSource());
+} else {
+  isReingest = false;
+}
 
 Review comment:
   For now, I think this config is best left undocumented and for auto 
compaction internal usage only.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

[GitBox]

himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335159299
 
 

 ##
 File path: core/src/main/java/org/apache/druid/timeline/CompactionState.java
 ##
 @@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.timeline;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.indexer.partitions.PartitionsSpec;
+
+import java.util.Map;
+import java.util.Objects;
+
+public class CompactionState
+{
+  private final PartitionsSpec partitionsSpec;
+  // org.apache.druid.segment.IndexSpec cannot be used here to avoid the 
dependency cycle
 
 Review comment:
   got it, thanks.
   
   maybe in next round of module merge: merge core into processing if there is 
no use case of anyone depending on druid-core directly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

[GitBox]

himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335159299
 
 

 ##
 File path: core/src/main/java/org/apache/druid/timeline/CompactionState.java
 ##
 @@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.timeline;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.indexer.partitions.PartitionsSpec;
+
+import java.util.Map;
+import java.util.Objects;
+
+public class CompactionState
+{
+  private final PartitionsSpec partitionsSpec;
+  // org.apache.druid.segment.IndexSpec cannot be used here to avoid the 
dependency cycle
 
 Review comment:
   got it, thanks.
   
   maybe in next round of module merge: merge core into processing if there is 
no use case of anyone depending on druid-core directly :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on a change in pull request #8573: Stateful auto compaction

[GitBox]

himanshug commented on a change in pull request #8573: Stateful auto compaction
URL: https://github.com/apache/incubator-druid/pull/8573#discussion_r335158395
 
 

 ##
 File path: core/src/main/java/org/apache/druid/timeline/CompactionState.java
 ##
 @@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.timeline;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import org.apache.druid.indexer.partitions.PartitionsSpec;
+
+import java.util.Map;
+import java.util.Objects;
+
+public class CompactionState
 
 Review comment:
   LGTM, thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] clintropolis commented on a change in pull request #8578: parallel broker merges on fork join pool

[GitBox]

clintropolis commented on a change in pull request #8578: parallel broker 
merges on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335153151
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
 
 Review comment:
   heh, thanks for reminding me to add some actually useful text here, I was 
also going to mention that functionally it is like a composite of 
`MergeSequence` and `CombiningSequence` :+1:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on a change in pull request #8578: parallel broker merges on fork join pool

[GitBox]

himanshug commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r335151691
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
 
 Review comment:
   not organic ? :-P
   
   maybe copy some part from 
https://github.com/apache/incubator-druid/issues/8577 also and add link to that 
.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on a change in pull request #8578: parallel broker merges on fork join pool

[GitBox]

himanshug commented on a change in pull request #8578: parallel broker merges 
on fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334687224
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new 

[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

[GitBox]

himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   Not saying that we build the "alert" mechanism itself (e.g. sending mail 
etc) .. but use existing emitter based Alert mechanism.
   Also, router endpoint could just return the checkup results from previous 
run making web-console experience very fast .
   
   
   makes sense ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

[GitBox]

himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   [Update] Not saying that we build the "alert" mechanism itself (e.g. sending 
mail etc) .. but use existing emitter based Alert mechanism.
   Also, router endpoint could just return the checkup results from previous 
run making web-console experience very fast .
   
   
   makes sense ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

[GitBox]

himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   [Update] Not saying that we build the "alert" mechanism itself (e.g. sending 
mail etc) .. but use existing emitter based Alert mechanism.
   Also, router endpoint could just return the checkup results from previous 
run making web-console experience very fast .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] leventov opened a new issue #8680: Make IntelliJ's inspection "Method is identical to its super method" a error

[GitBox]

leventov opened a new issue #8680: Make IntelliJ's inspection "Method is 
identical to its super method" a error
URL: https://github.com/apache/incubator-druid/issues/8680
 
 
   28 occurrences in the codebase, currently.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug edited a comment on issue #8672: Druid Doctor

[GitBox]

himanshug edited a comment on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   
   [Update] Not saying that we build the "alert" mechanism itself (e.g. sending 
mail etc) .. but use existing emitter based Alert mechanism.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on issue #8672: Druid Doctor

[GitBox]

himanshug commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542376613
 
 
   This is great, but as a followup It would be great if most of this is 
implemented on the router node with an endpoint , so web console would just 
call the endpoint to get results of checkup.
   
   advantage of that approach: same code at router could enable periodic 
checkups in background and automatically generate alerts for the user for many 
bad states proactively e.g. historical capacity is 90% utilized, more than x 
tasks are in pending state etc etc ... kind of things that we manually setup 
alerts for as operator of Druid cluster using other mechanisms. Building it 
inside Druid would help any user get a lot of that for free.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

[GitBox]

vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542365588
 
 
   @fjy are you reacting to something? Adjust if from what to what? Right now 
suggestions will show up in a neutral (dark) color and issues will show up in 
blueprint's warning style. You will get a red error if the tests early 
terminated due to an unrecoverable state (like in the screenshot I posted where 
the cluster is not even running).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] vogievetsky commented on issue #8674: Druid coordinator dynamic compaction failure

[GitBox]

vogievetsky commented on issue #8674: Druid coordinator dynamic compaction 
failure
URL: 
https://github.com/apache/incubator-druid/issues/8674#issuecomment-542363941
 
 
   What is you `druid.indexer.runner.type` set to?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly logging JsonMappingException when using druid-basic-security extension with an authenticator

[GitBox]

mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly 
logging JsonMappingException when using druid-basic-security extension with an 
authenticator that has no users setup
URL: 
https://github.com/apache/incubator-druid/issues/8678#issuecomment-542349688
 
 
   Hi @jon-wei and @nishantmonu51, Can one of you assign this ticket to me?
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] vaibhav-imply opened a new issue #8679: Feature request to support GREATEST/LEAST function in Druid sql

[GitBox]

vaibhav-imply opened a new issue #8679: Feature request to support  
GREATEST/LEAST  function in Druid sql 
URL: https://github.com/apache/incubator-druid/issues/8679
 
 
   ### Description
   Currently GREATEST/LEAST post-aggregators are available via native druid 
query. It will be great to have GREATEST/LEAST implementation of 
post-aggregators as Druid SQL functions.
   
   ### Motivation
   
   It will make user's life easier as they always feel comfortable writing an 
SQL over JSON query.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] fjy commented on issue #8672: Druid Doctor

[GitBox]

fjy commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542335390
 
 
   @vogievetsky can we adjust the color scheme to more accurately reflect the 
severity of an issue and minimize false scares? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] vogievetsky commented on issue #8672: Druid Doctor

[GitBox]

vogievetsky commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542331870
 
 
   @himanshug you (and anyone) can always help by providing me with more 
trouble shooting tests to encode!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] jihoonson commented on issue #8676: NumShards will cause broker can't load segments

[GitBox]

jihoonson commented on issue #8676: NumShards will cause broker can't load 
segments 
URL: 
https://github.com/apache/incubator-druid/issues/8676#issuecomment-542314018
 
 
   @licl2014 thank you for the report! I believe the same issue exists in 
master.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly logging JsonMappingException when using druid-basic-security extension with an authenticator

[GitBox]

mohammadjkhan commented on issue #8678: Non-coordinator services are repeatedly 
logging JsonMappingException when using druid-basic-security extension with an 
authenticator that has no users setup
URL: 
https://github.com/apache/incubator-druid/issues/8678#issuecomment-542313054
 
 
   I already have a fix for this issue and will be creating a PR shortly


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[incubator-druid-website-src] branch master updated: Clarify CLA policy.

[fjy]

This is an automated email from the ASF dual-hosted git repository.

fjy pushed a commit to branch master
in repository 
https://gitbox.apache.org/repos/asf/incubator-druid-website-src.git


The following commit(s) were added to refs/heads/master by this push:
 new d874b14  Clarify CLA policy.
 new 4e2ab97  Merge pull request #65 from gianm/cla
d874b14 is described below

commit d874b14e552db2a379d230499a9d4c1c9db8b56c
Author: Gian Merlino 
AuthorDate: Tue Oct 15 09:15:23 2019 -0700

Clarify CLA policy.
---
 community/cla.md   | 16 
 community/index.md |  4 
 2 files changed, 4 insertions(+), 16 deletions(-)

diff --git a/community/cla.md b/community/cla.md
index b2761c2..ed802cd 100644
--- a/community/cla.md
+++ b/community/cla.md
@@ -3,16 +3,8 @@ title: Druid Contributor License Agreement
 layout: simple_page
 ---
 
-For code contributions, we do require that a contributor license agreement
-(CLA) is agreed to before we can accept the code. Our CLA is fashioned after
-the Apache CLA.
+Per Apache policy, all Druid committers are required to have a contributor 
license agreement (CLA) on file with
+the Apache Software Foundation.
 
-### Corporate Agreement
-
-For corporate contributions, please fill out and return the [Druid Corporate 
CLA](https://docs.google.com/document/d/1fgAWpzJT5sVwXqSF9r00gMsoJlxZyw24Ifejb4oyyrc/edit?usp=sharing)
 to [druid-ad...@metamarkets.com](mailto:druid-ad...@metamarkets.com)
-
-### Individual Agreement
-
-For individual contributions please use the form below or return the [Druid 
Individual 
CLA](https://docs.google.com/document/d/1tHE41iQuQmowLgkBeAe7YMt6EfVjo2hLncvV80bfONs/edit?usp=sharing)
 to [druid-ad...@metamarkets.com](mailto:druid-ad...@metamarkets.com)
-
-https://docs.google.com/forms/d/11JzgE_sYVasOjgPVUu_lbjYnlA8n8cczEBGOyCfc9nE/viewform?embedded=true;
 width="760" height="1152" frameborder="0" marginheight="0" 
marginwidth="0">Loading...
+Other contributors are free to submit patches without having a CLA on file. 
You should only submit patches to the
+project when you intend to license your contribution under the Apache 2.0 
license, and have the legal right to do so.
diff --git a/community/index.md b/community/index.md
index 77a0c53..a8de0c9 100644
--- a/community/index.md
+++ b/community/index.md
@@ -76,10 +76,6 @@ In general please follow the [contributing 
guidelines](https://github.com/apache
 when sending in pull requests. This will help review proceed as quickly as
 possible.
 
-For code contributions, we require that you agree to a Contributor License
-Agreement (CLA) before we can accept your code. You can find our CLA on and
-sign it directly on our [CLA page](/community/cla.html)
-
 ### Committers
 
 Committers are collectively responsible for Druid's technical management. This 
involves


-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid-website-src] fjy merged pull request #65: Clarify CLA policy.

[GitBox]

fjy merged pull request #65: Clarify CLA policy.
URL: https://github.com/apache/incubator-druid-website-src/pull/65
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] mohammadjkhan opened a new issue #8678: Non-coordinator services are repeatedly logging JsonMappingException when using druid-basic-security extension with an authenticator

[GitBox]

mohammadjkhan opened a new issue #8678: Non-coordinator services are repeatedly 
logging JsonMappingException when using druid-basic-security extension with an 
authenticator that has no users setup
URL: https://github.com/apache/incubator-druid/issues/8678
 
 
   ### Affected Version
   Detected in master (0.17.0), but also occurs in versions 0.12.0 and above.
   
   ### Description
   Issue occurs when using druid-basic-security extension and having an 
authenticator in the authenticator chain that has no users setup (not even the 
admin and druid_system internal users). We're seeing the following exception 
getting logged repeatedly within non-coordinator services:
   WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 9) 
in 901ms.
   com.fasterxml.jackson.databind.JsonMappingException: No content to map due 
to end-of-input
   
   - Cluster size
   Any
   
   - Configurations in use
   // Coordinator config
   -server
   -Xms256m
   -Xmx256m
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=/tmp/druid
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
   -XX:+UseG1GC
   -Dlog4j.configurationFile=/path/to/log4j2.debug.xml
   -Dorg.jboss.logging.provider=slf4j
   -Ddruid.service=druid/coordinator
   -Ddruid.coordinator.period=PT10S
   -Ddruid.coordinator.startDelay=PT5S
   -Ddruid.host=localhost
   -Ddruid.extensions.directory=/path/to/apache/druid/extensions/
   
-Ddruid.extensions.loadList=[\"mysql-metadata-storage\",\"druid-basic-security\"]
   -Ddruid.zk.service.host=localhost
   -Ddruid.zk.paths.base=/druid
   -Ddruid.metadata.storage.type=mysql
   
-Ddruid.metadata.storage.connector.connectURI="jdbc:mysql://localhost:3306/druid"
   -Ddruid.metadata.storage.connector.user=druid1
   -Ddruid.metadata.storage.connector.password=test
   -Ddruid.emitter=logging
   -Ddruid.emitter.logging.logLevel=debug
   -Ddruid.sql.enable=true
   -Ddruid.auth.authenticator.local.type=basic
   -Ddruid.auth.authenticator.local.skipOnFailure=true
   -Ddruid.auth.authenticator.local.initialAdminPassword=password1
   -Ddruid.auth.authenticator.local.initialInternalClientPassword=password2
   -Ddruid.auth.authenticator.local.credentialsValidator.type=metadata
   -Ddruid.extensions.directory=/path/to/apache/druid/extensions/
   
-Ddruid.extensions.loadList=[\"mysql-metadata-storage\",\"druid-basic-security\"]
   -Ddruid.auth.authenticatorChain=[\"local\",\"test\"]
   -Ddruid.auth.authenticator.local.type=basic
   -Ddruid.auth.authenticator.local.skipOnFailure=true
   -Ddruid.auth.authenticator.local.initialAdminPassword=password1
   -Ddruid.auth.authenticator.local.initialInternalClientPassword=password2
   -Ddruid.auth.authenticator.local.credentialsValidator.type=metadata
   -Ddruid.auth.authenticator.local.authorizerName=local
   -Ddruid.auth.authenticator.test.type=basic
   -Ddruid.auth.authenticator.test.skipOnFailure=true
   -Ddruid.auth.authenticator.test.credentialsValidator.type=metadata
   -Ddruid.auth.authenticator.test.authorizerName=test
   -Ddruid.auth.authorizers=[\"local\",\"test\"]
   -Ddruid.auth.authorizer.local.type=basic
   -Ddruid.auth.authorizer.local.roleProvider.type=metadata
   -Ddruid.auth.authorizer.test.type=basic
   -Ddruid.auth.authorizer.test.roleProvider.type=metadata
   -Ddruid.escalator.type=basic
   -Ddruid.escalator.internalClientUsername=druid_system
   -Ddruid.escalator.internalClientPassword=password2
   -Ddruid.escalator.authorizerName=local
   
   // Broker config
   -server
   -Xms1g
   -Xmx1g
   -XX:MaxDirectMemorySize=1792m
   -Duser.timezone=UTC
   -Dfile.encoding=UTF-8
   -Djava.io.tmpdir=/tmp/druid
   -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
   -XX:+UseG1GC
   -Dlog4j.configurationFile=/path/to/log4j2.debug.xml
   -Dorg.jboss.logging.provider=slf4j
   -Ddruid.service=druid/broker
   -Ddruid.broker.http.numConnections=5
   -Ddruid.broker.cache.useCache=false
   -Ddruid.broker.cache.populateCache=false
   -Ddruid.server.http.numThreads=9
   -Ddruid.processing.buffer.sizeBytes=25600
   -Ddruid.processing.numThreads=2
   -Ddruid.cache.sizeInBytes=1000
   -Ddruid.host=localhost
   -Ddruid.extensions.directory=
   
-Ddruid.extensions.loadList=[\"mysql-metadata-storage\",\"druid-basic-security\"]
   -Ddruid.zk.service.host=localhost
   -Ddruid.zk.paths.base=/druid
   -Ddruid.metadata.storage.type=mysql
   
-Ddruid.metadata.storage.connector.connectURI="jdbc:mysql://localhost:3306/druid"
   -Ddruid.metadata.storage.connector.user=druid1
   -Ddruid.metadata.storage.connector.password=test
   -Ddruid.emitter=logging
   -Ddruid.emitter.logging.logLevel=debug
   -Ddruid.sql.enable=true
   -Ddruid.auth.authenticator.local.type=basic
   -Ddruid.auth.authenticator.local.skipOnFailure=true
   -Ddruid.auth.authenticator.local.initialAdminPassword=password1
   -Ddruid.auth.authenticator.local.initialInternalClientPassword=password2
   -Ddruid.auth.authenticator.local.credentialsValidator.type=metadata

[GitHub] [incubator-druid-website-src] gianm opened a new pull request #65: Clarify CLA policy.

[GitBox]

gianm opened a new pull request #65: Clarify CLA policy.
URL: https://github.com/apache/incubator-druid-website-src/pull/65
 
 
   Update the community & CLA pages to reflect Apache policy.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] himanshug commented on issue #8672: Druid Doctor

[GitBox]

himanshug commented on issue #8672: Druid Doctor
URL: https://github.com/apache/incubator-druid/pull/8672#issuecomment-542291030
 
 
   I am somewhat illiterate when it comes to modern UI frameworks and 
practices, so couldn't review the code but :+1:


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] teeram commented on issue #8663: Kafka indexing service duplicate entry exception in druid_pendingSegments

[GitBox]

teeram commented on issue #8663: Kafka indexing service duplicate entry 
exception in druid_pendingSegments
URL: 
https://github.com/apache/incubator-druid/issues/8663#issuecomment-542288337
 
 
   Hey @jihoonson, sorry for the delays in response. Hmm, I haven't turned on 
the minor compaction features mentioned in that issue, so I don't suspect that 
to be the culprit either.
   
   Unfortunately, I can't share those exact segment id's that are in conflict. 
But I am looking through those tables and can describe some of the oddities I 
am seeing. One thing I have noticed in the `druid_pendingSegments` table is 
that the id in conflict has a `created_date` that is typically from the 
previous day. For instance, I am getting an id conflict today (2019-10-15) for 
an entry in `druid_pendingSegments` with a `created_date` of 2019-10-14. That 
seems a little odd to me that the resumed Kafka indexing task would not attempt 
to place an entry in `druid_pendingSegments` with an updated value for the 
`created_date` field. There are other entries in the `druid_pendingSegments` 
table with a `created_date` field from today (2019-10-15).
   
   I did a quick query in the `druid_segments` table to see what segments were 
available for the time chunk in conflict. I found that all the segments in this 
table had been unpublished (`used = false`). This was the behavior I was 
expecting as I was dropping old segments before new ones were created.
   
   Thanks again for all your assistance!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] smokemonster99 commented on issue #4194: Kafka lag emitter - Kafka Indexing Service

[GitBox]

smokemonster99 commented on issue #4194: Kafka lag emitter - Kafka Indexing 
Service
URL: https://github.com/apache/incubator-druid/pull/4194#issuecomment-542287669
 
 
   @pjain1 thanks! That was it, I just needed to actually send something to 
kafka to get the metrics to show up in datadog via dogstatsd.
   
   Maybe you can help with this somewhat related question...My configured 
monitors such as jvm and sys work fine but I see there is a realtime monitor 
for ingestion, (org.apache.druid.segment.realtime.RealtimeMetricsMonitor) but 
when I configure it as a monitor on middlemanager, I get errors in the logs 
such as "No implementation for 
java.util.List was bound."
   
   Any ideas here? I am just looking for ingest metrics and thought that one 
was the best to use...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] gianm commented on issue #8673: InlineFirehose does not work with index_parallel ingestion

[GitBox]

gianm commented on issue #8673: InlineFirehose does not work with 
index_parallel ingestion
URL: 
https://github.com/apache/incubator-druid/issues/8673#issuecomment-542266767
 
 
   > If this gets fixed, is it possible to fix these cryptic messages that are 
sent out? I assume it's because of Jackson deserialization exceptions being 
regurgitated to the user?
   
   Yeah, those are Jackson exceptions, and it would be nice to make them more 
pleasant in a systematic way. I'm not sure if there's an easy way to do that 
without destroying the information in the error message or needing to parse 
their text. Maybe Jackson provides a way to get errors in a more programmatic 
way and we can write our own messages.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] sascha-coenen opened a new issue #8677: Misleading documentation/usage of "druid.indexing.doubleStorage" property

[GitBox]

sascha-coenen opened a new issue #8677: Misleading documentation/usage of 
"druid.indexing.doubleStorage" property
URL: https://github.com/apache/incubator-druid/issues/8677
 
 
   
   ### Affected Version
   
   0.11.x and following
   
   ### Description
   The release notes of Druid 0.11.0 
(https://github.com/apache/incubator-druid/releases/tag/druid-0.11.0) state 
that the "druid.indexing.doubleStorage" property could be specified in the 
common.runtime.properties file:
   "To enable Double column storage, set the following property in the common 
runtime properties"
   Furthermore, the Druid codebase contains example common.runtime.properties 
files in which this property is set:
   
https://github.com/apache/incubator-druid/blob/druid-0.16.0-incubating/examples/conf/druid/cluster/_common/common.runtime.properties
   
   However, both the Druid source code and also the Druid documentation seem to 
suggest that this property needs to be set as a java system property:
   in class ColumnHolder
   `
   static boolean storeDoubleAsFloat()
   {
 String value = System.getProperty(DOUBLE_STORAGE_TYPE_PROPERTY, "double");
 return !"double".equals(StringUtils.toLowerCase(value));
   }
   `
   
   Likewise the documentation correctly states:
   
https://github.com/apache/incubator-druid/blob/druid-0.16.0-incubating/docs/configuration/index.md
   
   "To keep the old format set the system-wide property 
druid.indexing.doubleStorage=float"
   
   
   In summary, this property only comes into effect if set as a java system 
property. Setting it within the commons.runtime.properties file has no effect.
   I propose to either remove the property from the common.runtime.properties 
example files and to correct the release notes OR to add support for specifying 
this property in the configuration files. I would prefer the later option.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] stale[bot] commented on issue #6790: chooseBestServer bug in CostBalancerStrategy

[GitBox]

stale[bot] commented on issue #6790: chooseBestServer bug in 
CostBalancerStrategy
URL: 
https://github.com/apache/incubator-druid/issues/6790#issuecomment-542224988
 
 
   This issue has been marked as stale due to 280 days of inactivity. It will 
be closed in 4 weeks if no further activity occurs. If this issue is still 
relevant, please simply write any comment. Even if closed, you can still revive 
the issue at any time or discuss it on the d...@druid.apache.org list. Thank 
you for your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] stale[bot] commented on issue #8098: Kafka lookup custom Json and Jq extractor

[GitBox]

stale[bot] commented on issue #8098: Kafka lookup custom Json and Jq extractor
URL: https://github.com/apache/incubator-druid/pull/8098#issuecomment-542224940
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 4 weeks if no further activity occurs. If you think that's 
incorrect or this pull request should instead be reviewed, please simply write 
any comment. Even if closed, you can still revive the PR at any time or discuss 
it on the d...@druid.apache.org list. Thank you for your contributions.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] jnaous commented on issue #8673: InlineFirehose does not work with index_parallel ingestion

[GitBox]

jnaous commented on issue #8673: InlineFirehose does not work with 
index_parallel ingestion
URL: 
https://github.com/apache/incubator-druid/issues/8673#issuecomment-542217699
 
 
   If this gets fixed, is it possible to fix these cryptic messages that are
   sent out? I assume it's because of Jackson deserialization exceptions being
   regurgitated to the user?
   
   On Mon, Oct 14, 2019 at 8:53 PM Jihoon Son  wrote:
   
   > I was writing about the same comment with @ccaominh
   > . InlineFirehose could implement
   > FiniteFirehoseFactory but it should always run in the sequential mode.
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
   
   -- 
   Jad Naous
   Imply | VP R
   650-521-3425
   jad.na...@imply.io
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] licl2014 opened a new issue #8676: NumShards will cause broker can't load segments

[GitBox]

licl2014 opened a new issue #8676: NumShards will cause broker can't load 
segments 
URL: https://github.com/apache/incubator-druid/issues/8676
 
 
   NumShards will cause broker can't load segments 
   
   ### Affected Version
   
   0.9.2/0.12.3
   
   ### Description
   
   
   - We load batch data which set `NumShards` to 100
   - The number of batch data is 10,  and their cardinality is 10 too.
   - So although the reduce number is 100, but finally the shards will only 10 
,and their shards number is not continuous.
   - Broker will add them in `incompletePartitionsTimeline` when creating 
versionedIntervalTimeline.
   - The query about this datasource will return `null`
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] leventov opened a new issue #8675: Document what does it mean for ServletFilterHolder.getDispatcherType() to return null

[GitBox]

leventov opened a new issue #8675: Document what does it mean for 
ServletFilterHolder.getDispatcherType() to return null
URL: https://github.com/apache/incubator-druid/issues/8675
 
 
   `ServletFilterHolder.getDispatcherType()` is annotated `@Nullable` but it 
doesn't explain in what cases it's OK (or not OK) to return null from this 
method for subclasses.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] mr0al opened a new issue #8674: Druid coordinator dynamic compaction failure

[GitBox]

mr0al opened a new issue #8674: Druid coordinator dynamic compaction failure
URL: https://github.com/apache/incubator-druid/issues/8674
 
 
   Druid 0.14.1
   
   I've set up druid dynamic compaction config on a realtime datasource but it 
seems this is even failing to launch a task. 
   If helps, I have long running indexer 5h with an intermediateHandoffPeriod 
of 5mins.
   Couldn't find anything anywhere on the below.. any ideas pls?
   
   ![Screen Shot 2019-10-15 at 10 06 49] 
(https://user-images.githubusercontent.com/5665595/66819286-a52d6980-ef36-11e9-92a3-1701b29dd083.png)
 
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org

[GitHub] [incubator-druid] leventov commented on a change in pull request #8578: parallel broker merges on fork join pool

[GitBox]

leventov commented on a change in pull request #8578: parallel broker merges on 
fork join pool
URL: https://github.com/apache/incubator-druid/pull/8578#discussion_r334845335
 
 

 ##
 File path: 
core/src/main/java/org/apache/druid/java/util/common/guava/ParallelMergeCombiningSequence.java
 ##
 @@ -0,0 +1,1071 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common.guava;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Ordering;
+import org.apache.druid.java.util.common.RE;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.utils.JvmUtils;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.Objects;
+import java.util.PriorityQueue;
+import java.util.Queue;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.RecursiveAction;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.BinaryOperator;
+
+/**
+ * Artisanal, locally-sourced, hand-crafted, gluten and GMO free, bespoke, 
small-batch parallel merge combinining sequence
+ */
+public class ParallelMergeCombiningSequence extends YieldingSequenceBase
+{
+  private static final Logger LOG = new 
Logger(ParallelMergeCombiningSequence.class);
+
+  private final ForkJoinPool workerPool;
+  private final List> baseSequences;
+  private final Ordering orderingFn;
+  private final BinaryOperator combineFn;
+  private final int queueSize;
+  private final boolean hasTimeout;
+  private final long timeoutAtNanos;
+  private final int queryPriority; // not currently used :(
+  private final int yieldAfter;
+  private final int batchSize;
+  private final int parallelism;
+  private final CancellationGizmo cancellationGizmo;
+
+  public ParallelMergeCombiningSequence(
+  ForkJoinPool workerPool,
+  List> baseSequences,
+  Ordering orderingFn,
+  BinaryOperator combineFn,
+  boolean hasTimeout,
+  long timeoutMillis,
+  int queryPriority,
+  int parallelism,
+  int yieldAfter,
+  int batchSize
+  )
+  {
+this.workerPool = workerPool;
+this.baseSequences = baseSequences;
+this.orderingFn = orderingFn;
+this.combineFn = combineFn;
+this.hasTimeout = hasTimeout;
+this.timeoutAtNanos = System.nanoTime() + 
TimeUnit.NANOSECONDS.convert(timeoutMillis, TimeUnit.MILLISECONDS);
+this.queryPriority = queryPriority;
+this.parallelism = parallelism;
+this.yieldAfter = yieldAfter;
+this.batchSize = batchSize;
+this.queueSize = 4 * (yieldAfter / batchSize);
+this.cancellationGizmo = new CancellationGizmo();
+  }
+
+  @Override
+  public  Yielder toYielder(OutType initValue, 
YieldingAccumulator accumulator)
+  {
+if (baseSequences.isEmpty()) {
+  return Sequences.empty().toYielder(initValue, accumulator);
+}
+
+final BlockingQueue> outputQueue = new 
ArrayBlockingQueue<>(queueSize);
+MergeCombinePartitioningAction finalMergeAction = new 
MergeCombinePartitioningAction<>(
+baseSequences,
+orderingFn,
+combineFn,
+outputQueue,
+queueSize,
+parallelism,
+yieldAfter,
+batchSize,
+hasTimeout,
+timeoutAtNanos,
+cancellationGizmo
+);
+workerPool.execute(finalMergeAction);
+Sequence finalOutSequence = makeOutputSequenceForQueue(outputQueue, 
hasTimeout, timeoutAtNanos, cancellationGizmo);
+return finalOutSequence.toYielder(initValue, accumulator);
+  }
+
+  /**
+   * Create an output {@link Sequence} that wraps the output {@link 
BlockingQueue} of a
+   * {@link MergeCombinePartitioningAction}
+   */
+  static  Sequence makeOutputSequenceForQueue(
+  BlockingQueue> queue,
+  boolean hasTimeout,
+  long timeoutAtNanos,
+  CancellationGizmo cancellationGizmo
+  )
+  {
+return new