[GitHub] [incubator-druid] jihoonson commented on issue #9013: Using annotation to distinguish Hadoop Configuration in each module
jihoonson commented on issue #9013: Using annotation to distinguish Hadoop Configuration in each module URL: https://github.com/apache/incubator-druid/pull/9013#issuecomment-564842692 @jnaous oh yeah, I will do it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] Maplejw opened a new issue #9016: druid dynamic configuration
Maplejw opened a new issue #9016: druid dynamic configuration
URL: https://github.com/apache/incubator-druid/issues/9016
The 'sw_template_behavior' datasource generates 24 segements each day. I
want to compact 24 to 1 segements every day dynamicly.
This is my config and datasource.
```curl
druid.coordinator.period.indexingPeriod=PT1800S
curl http://172.26.51.19:8081/druid/coordinator/v1/config/compaction
{
"compactionConfigs":[
{
"dataSource":"sw_template_behavior",
"taskPriority":25,
"inputSegmentSizeBytes":419430400,
"targetCompactionSizeBytes":419430400,
"maxRowsPerSegment":null,
"maxNumSegmentsToCompact":150,
"skipOffsetFromLatest":"P1D",
"tuningConfig":null,
"taskContext":null
}
],
"compactionTaskSlotRatio":0.2,
"maxCompactionTaskSlots":5
}
```
when 1800s comming,the log only shows
```log
org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentCompactor
- Found [1] available task slots for compaction out of [1] max compaction task
capacity
```
So I check the source code DruidCoordinatorSegmentCompactor.java.
iterator.hasNext() is false? I see the iterator is about exclue skip interval
segement. Do I set skipOffsetFromLatest wrong?
```code
for (; iterator.hasNext() && numSubmittedTasks <
numAvailableCompactionTaskSlots;) {
}
```
Please help me. Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson merged pull request #9015: Update architecture.md
jihoonson merged pull request #9015: Update architecture.md URL: https://github.com/apache/incubator-druid/pull/9015 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (66056b2 -> 13c33c1)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 66056b2 Using annotation to distinguish Hadoop Configuration in each module (#9013) add 13c33c1 Update architecture.md (#9015) No new revisions were added by this update. Summary of changes: docs/design/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jnaous commented on issue #9013: Using annotation to distinguish Hadoop Configuration in each module
jnaous commented on issue #9013: Using annotation to distinguish Hadoop Configuration in each module URL: https://github.com/apache/incubator-druid/pull/9013#issuecomment-564814785 @jihoonson should this be backported to the 0.17 branch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson merged pull request #9013: Using annotation to distinguish Hadoop Configuration in each module
jihoonson merged pull request #9013: Using annotation to distinguish Hadoop Configuration in each module URL: https://github.com/apache/incubator-druid/pull/9013 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on issue #9013: Using annotation to distinguish Hadoop Configuration in each module
jihoonson commented on issue #9013: Using annotation to distinguish Hadoop Configuration in each module URL: https://github.com/apache/incubator-druid/pull/9013#issuecomment-564809015 @himanshug @clintropolis thank you for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated: Using annotation to distinguish Hadoop Configuration in each module (#9013)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new 66056b2 Using annotation to distinguish Hadoop Configuration in each
module (#9013)
66056b2 is described below
commit 66056b282620e82fa464d79b0f76b1687b92530e
Author: Jihoon Son
AuthorDate: Wed Dec 11 17:30:44 2019 -0800
Using annotation to distinguish Hadoop Configuration in each module (#9013)
* Multibinding for NodeRole
* Fix endpoints
* fix doc
* fix test
* Using annotation to distinguish Hadoop Configuration in each module
---
.../druid/firehose/hdfs/HdfsFirehoseFactory.java | 3 +-
.../src/main/java/org/apache/druid/guice/Hdfs.java | 40 ++
.../druid/inputsource/hdfs/HdfsInputSource.java| 3 +-
.../druid/storage/hdfs/HdfsDataSegmentKiller.java | 3 +-
.../druid/storage/hdfs/HdfsDataSegmentPuller.java | 3 +-
.../druid/storage/hdfs/HdfsDataSegmentPusher.java | 7 +++-
.../hdfs/HdfsFileTimestampVersionFinder.java | 3 +-
.../storage/hdfs/HdfsStorageAuthentication.java| 3 +-
.../druid/storage/hdfs/HdfsStorageDruidModule.java | 4 +--
.../druid/storage/hdfs/tasklog/HdfsTaskLogs.java | 3 +-
.../druid/data/input/orc/OrcExtensionsModule.java | 3 +-
.../druid/data/input/orc/OrcInputFormat.java | 3 +-
.../org/apache/druid/data/input/orc/guice/Orc.java | 40 ++
.../input/parquet/ParquetExtensionsModule.java | 3 +-
.../data/input/parquet/ParquetInputFormat.java | 10 --
.../druid/data/input/parquet/ParquetReader.java| 6 +++-
.../druid/data/input/parquet/guice/Parquet.java| 40 ++
.../data/input/parquet/BaseParquetReaderTest.java | 3 +-
18 files changed, 163 insertions(+), 17 deletions(-)
diff --git
a/extensions-core/hdfs-storage/src/main/java/org/apache/druid/firehose/hdfs/HdfsFirehoseFactory.java
b/extensions-core/hdfs-storage/src/main/java/org/apache/druid/firehose/hdfs/HdfsFirehoseFactory.java
index 962da5e..ee5bf03 100644
---
a/extensions-core/hdfs-storage/src/main/java/org/apache/druid/firehose/hdfs/HdfsFirehoseFactory.java
+++
b/extensions-core/hdfs-storage/src/main/java/org/apache/druid/firehose/hdfs/HdfsFirehoseFactory.java
@@ -27,6 +27,7 @@ import org.apache.druid.data.input.FiniteFirehoseFactory;
import org.apache.druid.data.input.InputSplit;
import org.apache.druid.data.input.impl.StringInputRowParser;
import
org.apache.druid.data.input.impl.prefetch.PrefetchableTextFilesFirehoseFactory;
+import org.apache.druid.guice.Hdfs;
import org.apache.druid.inputsource.hdfs.HdfsInputSource;
import org.apache.druid.storage.hdfs.HdfsDataSegmentPuller;
import org.apache.druid.utils.CompressionUtils;
@@ -46,7 +47,7 @@ public class HdfsFirehoseFactory extends
PrefetchableTextFilesFirehoseFactoryhttp://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.guice;
+
+import com.google.inject.BindingAnnotation;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * Each extension module needs to properly bind whatever it will use, but
sometimes different modules need to bind the
+ * same class which will lead to the duplicate injection error. To avoid this
problem, each module is supposed to bind
+ * different instances. This is a binding annotation for druid-hdfs-storage
extension. Any binding for the same type
+ * should be distinguished by using this annotation.
+ */
+@Target({ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD})
+@Retention(RetentionPolicy.RUNTIME)
+@BindingAnnotation
+public @interface Hdfs
+{
+}
diff --git
a/extensions-core/hdfs-storage/src/main/java/org/apache/druid/inputsource/hdfs/HdfsInputSource.java
b/extensions-core/hdfs-storage/src/main/java/org/apache/druid/inputsource/hdfs/HdfsInputSource.java
index 1409aa6..fe4a18b 100644
---
a/extensions-core/hdfs-storage/src/main/java/org/apache/druid/inputsource/hdfs/HdfsInputSource.java
+++
b/extensions-core/hdfs-storage/src/main/java/org/apache/druid/inputsource/hdfs/HdfsInputSource.java
@@ -32,6 +32,7 @@ import org.apache.druid.data.input.InputSplit;
import org.apache.druid.data.input.SplitHintSpec;
import org.apache.druid.data.input.impl.InputEntityIteratingReader;
import org.apache.druid.data.input.impl.SplittableInputSource;
+import org.apache.druid.guice.Hdfs;
import org.apache.druid.java.util.common.IAE;
[GitHub] [incubator-druid] jihoonson closed issue #9004: Hadoop task fails because of duplicate injection of Configuration
jihoonson closed issue #9004: Hadoop task fails because of duplicate injection of Configuration URL: https://github.com/apache/incubator-druid/issues/9004 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
jihoonson commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356919309
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1021,92 +1043,164 @@ public void unregisterListener(String listenerId)
HttpRemoteTaskRunnerWorkItem.State.PENDING
);
tasks.put(task.getId(), taskRunnerWorkItem);
-addPendingTaskToExecutor(task.getId());
+pendingTaskIds.add(task.getId());
+
+statusLock.notifyAll();
+
return taskRunnerWorkItem.getResult();
}
}
}
- private void addPendingTaskToExecutor(final String taskId)
+ private void startPendingTaskHandling()
{
-pendingTasksExec.execute(
-() -> {
- while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
-ImmutableWorkerInfo immutableWorker;
-HttpRemoteTaskRunnerWorkItem taskItem = null;
+for (int i = 0; i < config.getPendingTasksRunnerNumThreads(); i++) {
+ pendingTasksExec.submit(
+ () -> {
try {
- synchronized (statusLock) {
-taskItem = tasks.get(taskId);
+ if (!lifecycleLock.awaitStarted()) {
+log.makeAlert("Lifecycle not started, PendingTaskExecution
loop will not run.").emit();
+return;
+ }
-if (taskItem == null) {
- log.info(
- "Task[%s] work item not found. Probably user asked to
shutdown before. Not assigning.",
- taskId
- );
- return;
-}
+ pendingTasksExecutionLoop();
+}
+catch (Throwable t) {
+ log.makeAlert(t, "Error while waiting for lifecycle start.
PendingTaskExecution loop will not run")
+ .emit();
+}
+finally {
+ log.info("PendingTaskExecution loop exited.");
+}
+ }
+ );
+}
+ }
-if (taskItem.getState() !=
HttpRemoteTaskRunnerWorkItem.State.PENDING) {
- log.info(
- "Task[%s] is in state[%s]. Probably some worker already
reported it. Not assigning.",
- taskId,
- taskItem.getState()
- );
- return;
-}
+ private void pendingTasksExecutionLoop()
+ {
+while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
+ try {
+// Find one pending task to run and a worker to run on
+HttpRemoteTaskRunnerWorkItem taskItem = null;
+ImmutableWorkerInfo immutableWorker = null;
-if (taskItem.getTask() == null) {
- throw new ISE("WTF! couldn't find Task instance for
taskId[%s].", taskId);
-}
-immutableWorker = findWorkerToRunTask(taskItem.getTask());
-
-if (immutableWorker == null) {
- // no free worker, wait for some worker to become free
-
statusLock.wait(config.getWaitForWorkerSlot().toStandardDuration().getMillis());
- continue;
-} else if (workersWithUnacknowledgedTask.putIfAbsent(
-immutableWorker.getWorker().getHost(),
-taskId
-) != null) {
- // there was a race and someone else took this worker slot,
try again
- continue;
-}
- }
+synchronized (statusLock) {
+ Iterator iter = pendingTaskIds.iterator();
+ while (iter.hasNext()) {
+String taskId = iter.next();
+HttpRemoteTaskRunnerWorkItem ti = tasks.get(taskId);
+
+if (ti == null || !ti.getState().isPending()) {
+ // happens if the task was shutdown or was picked up earlier and
no more pending
+ iter.remove();
Review comment:
Hmm, sorry I don't think I fully understand this logic yet.
> before leaving the lock , ti state is updated to PENDING_WORKER_ASSIGN so
no other task executor thread can pick up.
Do you mean the other threads can pick up even after the ti state is set to
`PENDING_WORKER_ASSIGN` but will ignore it in the [below if
clause](https://github.com/apache/incubator-druid/pull/8697/files#diff-b994d2f0b67b07608a060f52a17fbd01R1107)?
Or, do they really not pick up because somehow `pendingTaskIds` is updated
before other threads start picking up?
> `pendingTaskIds` variable declaration has a comment that this variable is
exclusively manipulated by only external task submitter threads or task
executor threads which I preferred.
Yes, I
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
jihoonson commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356919345
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1021,92 +1043,164 @@ public void unregisterListener(String listenerId)
HttpRemoteTaskRunnerWorkItem.State.PENDING
);
tasks.put(task.getId(), taskRunnerWorkItem);
-addPendingTaskToExecutor(task.getId());
+pendingTaskIds.add(task.getId());
+
+statusLock.notifyAll();
+
return taskRunnerWorkItem.getResult();
}
}
}
- private void addPendingTaskToExecutor(final String taskId)
+ private void startPendingTaskHandling()
{
-pendingTasksExec.execute(
-() -> {
- while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
-ImmutableWorkerInfo immutableWorker;
-HttpRemoteTaskRunnerWorkItem taskItem = null;
+for (int i = 0; i < config.getPendingTasksRunnerNumThreads(); i++) {
+ pendingTasksExec.submit(
+ () -> {
try {
- synchronized (statusLock) {
-taskItem = tasks.get(taskId);
+ if (!lifecycleLock.awaitStarted()) {
+log.makeAlert("Lifecycle not started, PendingTaskExecution
loop will not run.").emit();
+return;
+ }
-if (taskItem == null) {
- log.info(
- "Task[%s] work item not found. Probably user asked to
shutdown before. Not assigning.",
- taskId
- );
- return;
-}
+ pendingTasksExecutionLoop();
+}
+catch (Throwable t) {
+ log.makeAlert(t, "Error while waiting for lifecycle start.
PendingTaskExecution loop will not run")
+ .emit();
+}
+finally {
+ log.info("PendingTaskExecution loop exited.");
+}
+ }
+ );
+}
+ }
-if (taskItem.getState() !=
HttpRemoteTaskRunnerWorkItem.State.PENDING) {
- log.info(
- "Task[%s] is in state[%s]. Probably some worker already
reported it. Not assigning.",
- taskId,
- taskItem.getState()
- );
- return;
-}
+ private void pendingTasksExecutionLoop()
+ {
+while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
+ try {
+// Find one pending task to run and a worker to run on
+HttpRemoteTaskRunnerWorkItem taskItem = null;
+ImmutableWorkerInfo immutableWorker = null;
-if (taskItem.getTask() == null) {
- throw new ISE("WTF! couldn't find Task instance for
taskId[%s].", taskId);
-}
-immutableWorker = findWorkerToRunTask(taskItem.getTask());
-
-if (immutableWorker == null) {
- // no free worker, wait for some worker to become free
-
statusLock.wait(config.getWaitForWorkerSlot().toStandardDuration().getMillis());
- continue;
-} else if (workersWithUnacknowledgedTask.putIfAbsent(
-immutableWorker.getWorker().getHost(),
-taskId
-) != null) {
- // there was a race and someone else took this worker slot,
try again
- continue;
-}
- }
+synchronized (statusLock) {
+ Iterator iter = pendingTaskIds.iterator();
+ while (iter.hasNext()) {
+String taskId = iter.next();
+HttpRemoteTaskRunnerWorkItem ti = tasks.get(taskId);
+
+if (ti == null || !ti.getState().isPending()) {
+ // happens if the task was shutdown or was picked up earlier and
no more pending
+ iter.remove();
+ continue;
+}
- try {
-// this will send HTTP request to worker for assigning task
and hence kept
-// outside the synchronized block.
-if (runTaskOnWorker(taskItem,
immutableWorker.getWorker().getHost())) {
- return;
-}
- }
- finally {
-
workersWithUnacknowledgedTask.remove(immutableWorker.getWorker().getHost());
-synchronized (statusLock) {
- statusLock.notifyAll();
-}
- }
+if (ti.getState() ==
HttpRemoteTaskRunnerWorkItem.State.PENDING_WORKER_ASSIGN) {
+ // picked up by another pending task executor thread.
+ continue;
[GitHub] [incubator-druid] jihoonson commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
jihoonson commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356819299
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -313,6 +325,25 @@ private void scheduleCompletedTaskStatusCleanupFromZk()
);
}
+ /**
+ * Must not be used outside of this class and {@link
HttpRemoteTaskRunnerResource}
+ */
+ @SuppressWarnings("GuardedBy") // Read on workersWithUnacknowledgedTask is
safe
+ public Map getWorkersEligibleToRunTasks()
+ {
+return Maps.transformEntries(
+Maps.filterEntries(
+workers,
+input -> !lazyWorkers.containsKey(input.getKey()) &&
+
!workersWithUnacknowledgedTask.containsKey(input.getKey()) &&
+ !blackListedWorkers.containsKey(input.getKey()) &&
Review comment:
Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] benhopp opened a new pull request #9015: Update architecture.md
benhopp opened a new pull request #9015: Update architecture.md URL: https://github.com/apache/incubator-druid/pull/9015 fixed typo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson merged pull request #9005: Fix broken master
jihoonson merged pull request #9005: Fix broken master URL: https://github.com/apache/incubator-druid/pull/9005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (8af41d7 -> e5e1e9c)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 8af41d7 Update version to 0.18.0-incubating-SNAPSHOT (#9009) add e5e1e9c Fix broken master (#9005) No new revisions were added by this update. Summary of changes: docs/operations/api-reference.md | 4 +- .../druid/server/http/SelfDiscoveryResource.java | 57 +- .../http/security/SecurityResourceFilterTest.java | 5 +- .../main/java/org/apache/druid/cli/CliBroker.java | 5 +- .../java/org/apache/druid/cli/CliCoordinator.java | 5 +- .../java/org/apache/druid/cli/CliHistorical.java | 5 +- .../main/java/org/apache/druid/cli/CliIndexer.java | 2 +- .../org/apache/druid/cli/CliMiddleManager.java | 4 +- .../java/org/apache/druid/cli/CliOverlord.java | 5 +- .../main/java/org/apache/druid/cli/CliRouter.java | 7 +-- .../java/org/apache/druid/cli/ServerRunnable.java | 33 - 11 files changed, 77 insertions(+), 55 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] jihoonson closed issue #9002: Broken master because of duplicate injection
jihoonson closed issue #9002: Broken master because of duplicate injection URL: https://github.com/apache/incubator-druid/issues/9002 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8658: [Lookup] LookupExtractorFactoryMapContainer will cause jdbc password leak issue
himanshug commented on issue #8658: [Lookup] LookupExtractorFactoryMapContainer will cause jdbc password leak issue URL: https://github.com/apache/incubator-druid/issues/8658#issuecomment-564756727 I agree with @jon-wei , to protect the password using `EnvironmentVariablePasswordProvider` or your own implementation to `PasswordProvider` would be the right approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[incubator-druid] branch master updated (24fe824 -> 8af41d7)
This is an automated email from the ASF dual-hosted git repository. himanshug pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-druid.git. from 24fe824 add readiness endpoints to processes having initialization delays (#8841) add 8af41d7 Update version to 0.18.0-incubating-SNAPSHOT (#9009) No new revisions were added by this update. Summary of changes: benchmarks/pom.xml | 2 +- cloud/aws-common/pom.xml | 2 +- cloud/gcp-common/pom.xml | 2 +- core/pom.xml | 2 +- distribution/pom.xml | 2 +- extendedset/pom.xml | 2 +- extensions-contrib/ambari-metrics-emitter/pom.xml| 2 +- extensions-contrib/azure-extensions/pom.xml | 2 +- extensions-contrib/cassandra-storage/pom.xml | 2 +- extensions-contrib/cloudfiles-extensions/pom.xml | 2 +- extensions-contrib/distinctcount/pom.xml | 2 +- extensions-contrib/dropwizard-emitter/pom.xml| 2 +- extensions-contrib/graphite-emitter/pom.xml | 2 +- extensions-contrib/influx-extensions/pom.xml | 2 +- extensions-contrib/influxdb-emitter/pom.xml | 2 +- extensions-contrib/kafka-emitter/pom.xml | 2 +- extensions-contrib/materialized-view-maintenance/pom.xml | 2 +- extensions-contrib/materialized-view-selection/pom.xml | 2 +- extensions-contrib/momentsketch/pom.xml | 2 +- extensions-contrib/moving-average-query/pom.xml | 2 +- extensions-contrib/opentsdb-emitter/pom.xml | 2 +- extensions-contrib/redis-cache/pom.xml | 2 +- extensions-contrib/sqlserver-metadata-storage/pom.xml| 2 +- extensions-contrib/statsd-emitter/pom.xml| 2 +- extensions-contrib/tdigestsketch/pom.xml | 2 +- extensions-contrib/thrift-extensions/pom.xml | 2 +- extensions-contrib/time-min-max/pom.xml | 2 +- extensions-contrib/virtual-columns/pom.xml | 2 +- extensions-core/avro-extensions/pom.xml | 2 +- extensions-core/datasketches/pom.xml | 2 +- extensions-core/druid-basic-security/pom.xml | 2 +- extensions-core/druid-bloom-filter/pom.xml | 2 +- extensions-core/druid-kerberos/pom.xml | 2 +- extensions-core/ec2-extensions/pom.xml | 2 +- extensions-core/google-extensions/pom.xml| 2 +- extensions-core/hdfs-storage/pom.xml | 2 +- extensions-core/histogram/pom.xml| 2 +- extensions-core/kafka-extraction-namespace/pom.xml | 2 +- extensions-core/kafka-indexing-service/pom.xml | 2 +- extensions-core/kinesis-indexing-service/pom.xml | 2 +- extensions-core/lookups-cached-global/pom.xml| 2 +- extensions-core/lookups-cached-single/pom.xml| 2 +- extensions-core/mysql-metadata-storage/pom.xml | 2 +- extensions-core/orc-extensions/pom.xml | 2 +- extensions-core/parquet-extensions/pom.xml | 2 +- extensions-core/postgresql-metadata-storage/pom.xml | 2 +- extensions-core/protobuf-extensions/pom.xml | 2 +- extensions-core/s3-extensions/pom.xml| 2 +- extensions-core/simple-client-sslcontext/pom.xml | 2 +- extensions-core/stats/pom.xml| 2 +- hll/pom.xml | 2 +- indexing-hadoop/pom.xml | 2 +- indexing-service/pom.xml | 2 +- integration-tests/pom.xml| 2 +- pom.xml | 2 +- processing/pom.xml | 2 +- server/pom.xml | 2 +- services/pom.xml | 2 +- sql/pom.xml | 2 +- web-console/pom.xml | 2 +- website/pom.xml | 2 +- 61 files changed, 61 insertions(+), 61 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug merged pull request #9009: Update version to 0.18.0-incubating-SNAPSHOT
himanshug merged pull request #9009: Update version to 0.18.0-incubating-SNAPSHOT URL: https://github.com/apache/incubator-druid/pull/9009 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #9010: optionally enable Jetty ForwardedRequestCustomizer
himanshug commented on issue #9010: optionally enable Jetty ForwardedRequestCustomizer URL: https://github.com/apache/incubator-druid/pull/9010#issuecomment-564710698 added to 0.17.0 as it is pretty simple and I am already using it, will save me a backport on the custom druid build we deploy when we migrate to 0.17.0 . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on issue #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on issue #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots URL: https://github.com/apache/incubator-druid/pull/8697#issuecomment-564706872 @jihoonson thanks for reviewing and you are right about LGTM alert ... I completely missed that it was used in finally block as well. moved the null check out of that try-catch . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356803046
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1021,92 +1043,164 @@ public void unregisterListener(String listenerId)
HttpRemoteTaskRunnerWorkItem.State.PENDING
);
tasks.put(task.getId(), taskRunnerWorkItem);
-addPendingTaskToExecutor(task.getId());
+pendingTaskIds.add(task.getId());
+
+statusLock.notifyAll();
+
return taskRunnerWorkItem.getResult();
}
}
}
- private void addPendingTaskToExecutor(final String taskId)
+ private void startPendingTaskHandling()
{
-pendingTasksExec.execute(
-() -> {
- while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
-ImmutableWorkerInfo immutableWorker;
-HttpRemoteTaskRunnerWorkItem taskItem = null;
+for (int i = 0; i < config.getPendingTasksRunnerNumThreads(); i++) {
+ pendingTasksExec.submit(
+ () -> {
try {
- synchronized (statusLock) {
-taskItem = tasks.get(taskId);
+ if (!lifecycleLock.awaitStarted()) {
+log.makeAlert("Lifecycle not started, PendingTaskExecution
loop will not run.").emit();
+return;
+ }
-if (taskItem == null) {
- log.info(
- "Task[%s] work item not found. Probably user asked to
shutdown before. Not assigning.",
- taskId
- );
- return;
-}
+ pendingTasksExecutionLoop();
+}
+catch (Throwable t) {
+ log.makeAlert(t, "Error while waiting for lifecycle start.
PendingTaskExecution loop will not run")
+ .emit();
+}
+finally {
+ log.info("PendingTaskExecution loop exited.");
+}
+ }
+ );
+}
+ }
-if (taskItem.getState() !=
HttpRemoteTaskRunnerWorkItem.State.PENDING) {
- log.info(
- "Task[%s] is in state[%s]. Probably some worker already
reported it. Not assigning.",
- taskId,
- taskItem.getState()
- );
- return;
-}
+ private void pendingTasksExecutionLoop()
+ {
+while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
+ try {
+// Find one pending task to run and a worker to run on
+HttpRemoteTaskRunnerWorkItem taskItem = null;
+ImmutableWorkerInfo immutableWorker = null;
-if (taskItem.getTask() == null) {
- throw new ISE("WTF! couldn't find Task instance for
taskId[%s].", taskId);
-}
-immutableWorker = findWorkerToRunTask(taskItem.getTask());
-
-if (immutableWorker == null) {
- // no free worker, wait for some worker to become free
-
statusLock.wait(config.getWaitForWorkerSlot().toStandardDuration().getMillis());
- continue;
-} else if (workersWithUnacknowledgedTask.putIfAbsent(
-immutableWorker.getWorker().getHost(),
-taskId
-) != null) {
- // there was a race and someone else took this worker slot,
try again
- continue;
-}
- }
+synchronized (statusLock) {
+ Iterator iter = pendingTaskIds.iterator();
+ while (iter.hasNext()) {
+String taskId = iter.next();
+HttpRemoteTaskRunnerWorkItem ti = tasks.get(taskId);
+
+if (ti == null || !ti.getState().isPending()) {
+ // happens if the task was shutdown or was picked up earlier and
no more pending
+ iter.remove();
Review comment:
before leaving the lock , ti state is updated to PENDING_WORKER_ASSIGN so no
other task executor thread can pick up.
`pendingTaskIds` variable declaration has a comment that this variable is
exclusively manipulated by only external task submitter threads or task
executor threads which I preferred.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail:
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356801359
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1021,92 +1043,164 @@ public void unregisterListener(String listenerId)
HttpRemoteTaskRunnerWorkItem.State.PENDING
);
tasks.put(task.getId(), taskRunnerWorkItem);
-addPendingTaskToExecutor(task.getId());
+pendingTaskIds.add(task.getId());
+
+statusLock.notifyAll();
+
return taskRunnerWorkItem.getResult();
}
}
}
- private void addPendingTaskToExecutor(final String taskId)
+ private void startPendingTaskHandling()
{
-pendingTasksExec.execute(
-() -> {
- while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
-ImmutableWorkerInfo immutableWorker;
-HttpRemoteTaskRunnerWorkItem taskItem = null;
+for (int i = 0; i < config.getPendingTasksRunnerNumThreads(); i++) {
+ pendingTasksExec.submit(
+ () -> {
try {
- synchronized (statusLock) {
-taskItem = tasks.get(taskId);
+ if (!lifecycleLock.awaitStarted()) {
+log.makeAlert("Lifecycle not started, PendingTaskExecution
loop will not run.").emit();
+return;
+ }
-if (taskItem == null) {
- log.info(
- "Task[%s] work item not found. Probably user asked to
shutdown before. Not assigning.",
- taskId
- );
- return;
-}
+ pendingTasksExecutionLoop();
+}
+catch (Throwable t) {
+ log.makeAlert(t, "Error while waiting for lifecycle start.
PendingTaskExecution loop will not run")
+ .emit();
+}
+finally {
+ log.info("PendingTaskExecution loop exited.");
+}
+ }
+ );
+}
+ }
-if (taskItem.getState() !=
HttpRemoteTaskRunnerWorkItem.State.PENDING) {
- log.info(
- "Task[%s] is in state[%s]. Probably some worker already
reported it. Not assigning.",
- taskId,
- taskItem.getState()
- );
- return;
-}
+ private void pendingTasksExecutionLoop()
+ {
+while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
+ try {
+// Find one pending task to run and a worker to run on
+HttpRemoteTaskRunnerWorkItem taskItem = null;
+ImmutableWorkerInfo immutableWorker = null;
-if (taskItem.getTask() == null) {
- throw new ISE("WTF! couldn't find Task instance for
taskId[%s].", taskId);
-}
-immutableWorker = findWorkerToRunTask(taskItem.getTask());
-
-if (immutableWorker == null) {
- // no free worker, wait for some worker to become free
-
statusLock.wait(config.getWaitForWorkerSlot().toStandardDuration().getMillis());
- continue;
-} else if (workersWithUnacknowledgedTask.putIfAbsent(
-immutableWorker.getWorker().getHost(),
-taskId
-) != null) {
- // there was a race and someone else took this worker slot,
try again
- continue;
-}
- }
+synchronized (statusLock) {
+ Iterator iter = pendingTaskIds.iterator();
+ while (iter.hasNext()) {
+String taskId = iter.next();
+HttpRemoteTaskRunnerWorkItem ti = tasks.get(taskId);
+
+if (ti == null || !ti.getState().isPending()) {
+ // happens if the task was shutdown or was picked up earlier and
no more pending
+ iter.remove();
+ continue;
+}
- try {
-// this will send HTTP request to worker for assigning task
and hence kept
-// outside the synchronized block.
-if (runTaskOnWorker(taskItem,
immutableWorker.getWorker().getHost())) {
- return;
-}
- }
- finally {
-
workersWithUnacknowledgedTask.remove(immutableWorker.getWorker().getHost());
-synchronized (statusLock) {
- statusLock.notifyAll();
-}
- }
+if (ti.getState() ==
HttpRemoteTaskRunnerWorkItem.State.PENDING_WORKER_ASSIGN) {
+ // picked up by another pending task executor thread.
+ continue;
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356801301
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1021,92 +1043,164 @@ public void unregisterListener(String listenerId)
HttpRemoteTaskRunnerWorkItem.State.PENDING
);
tasks.put(task.getId(), taskRunnerWorkItem);
-addPendingTaskToExecutor(task.getId());
+pendingTaskIds.add(task.getId());
+
+statusLock.notifyAll();
+
return taskRunnerWorkItem.getResult();
}
}
}
- private void addPendingTaskToExecutor(final String taskId)
+ private void startPendingTaskHandling()
{
-pendingTasksExec.execute(
-() -> {
- while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
-ImmutableWorkerInfo immutableWorker;
-HttpRemoteTaskRunnerWorkItem taskItem = null;
+for (int i = 0; i < config.getPendingTasksRunnerNumThreads(); i++) {
+ pendingTasksExec.submit(
+ () -> {
try {
- synchronized (statusLock) {
-taskItem = tasks.get(taskId);
+ if (!lifecycleLock.awaitStarted()) {
+log.makeAlert("Lifecycle not started, PendingTaskExecution
loop will not run.").emit();
+return;
+ }
-if (taskItem == null) {
- log.info(
- "Task[%s] work item not found. Probably user asked to
shutdown before. Not assigning.",
- taskId
- );
- return;
-}
+ pendingTasksExecutionLoop();
+}
+catch (Throwable t) {
+ log.makeAlert(t, "Error while waiting for lifecycle start.
PendingTaskExecution loop will not run")
+ .emit();
+}
+finally {
+ log.info("PendingTaskExecution loop exited.");
+}
+ }
+ );
+}
+ }
-if (taskItem.getState() !=
HttpRemoteTaskRunnerWorkItem.State.PENDING) {
- log.info(
- "Task[%s] is in state[%s]. Probably some worker already
reported it. Not assigning.",
- taskId,
- taskItem.getState()
- );
- return;
-}
+ private void pendingTasksExecutionLoop()
+ {
+while (!Thread.interrupted() && lifecycleLock.awaitStarted(1,
TimeUnit.MILLISECONDS)) {
+ try {
+// Find one pending task to run and a worker to run on
+HttpRemoteTaskRunnerWorkItem taskItem = null;
+ImmutableWorkerInfo immutableWorker = null;
-if (taskItem.getTask() == null) {
- throw new ISE("WTF! couldn't find Task instance for
taskId[%s].", taskId);
-}
-immutableWorker = findWorkerToRunTask(taskItem.getTask());
-
-if (immutableWorker == null) {
- // no free worker, wait for some worker to become free
-
statusLock.wait(config.getWaitForWorkerSlot().toStandardDuration().getMillis());
- continue;
-} else if (workersWithUnacknowledgedTask.putIfAbsent(
-immutableWorker.getWorker().getHost(),
-taskId
-) != null) {
- // there was a race and someone else took this worker slot,
try again
- continue;
-}
- }
+synchronized (statusLock) {
+ Iterator iter = pendingTaskIds.iterator();
+ while (iter.hasNext()) {
+String taskId = iter.next();
+HttpRemoteTaskRunnerWorkItem ti = tasks.get(taskId);
+
+if (ti == null || !ti.getState().isPending()) {
+ // happens if the task was shutdown or was picked up earlier and
no more pending
+ iter.remove();
+ continue;
+}
- try {
-// this will send HTTP request to worker for assigning task
and hence kept
-// outside the synchronized block.
-if (runTaskOnWorker(taskItem,
immutableWorker.getWorker().getHost())) {
- return;
-}
- }
- finally {
-
workersWithUnacknowledgedTask.remove(immutableWorker.getWorker().getHost());
-synchronized (statusLock) {
- statusLock.notifyAll();
-}
- }
+if (ti.getState() ==
HttpRemoteTaskRunnerWorkItem.State.PENDING_WORKER_ASSIGN) {
+ // picked up by another pending task executor thread.
+ continue;
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356800868
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1501,6 +1623,23 @@ public void setState(State state)
state
);
+ setStateUnconditionally(state);
+}
+
+public void revertStateFromPendingWorkerAssignToPending()
+{
+ Preconditions.checkState(
+ this.state == State.PENDING_WORKER_ASSIGN,
+ "Can't move state from [%s] to [%s]",
+ this.state,
+ State.PENDING
+ );
+
+ setStateUnconditionally(State.PENDING);
+}
+
+private void setStateUnconditionally(State state)
+{
if (log.isDebugEnabled()) {
log.debug(
new RuntimeException("Stacktrace..."),
Review comment:
yes, added a comment too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356800295
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1432,15 +1533,35 @@ void taskAddedOrUpdated(final TaskAnnouncement
announcement, final WorkerHolder
{
enum State
{
- PENDING(0),
- RUNNING(1),
- COMPLETE(2);
+ // Task has been given to HRTR, but a worker to run this task hasn't
been identified yet.
+ PENDING(0, true, RunnerTaskState.PENDING),
+
+ // A Worker has been identified to run this task, but request to run
task hasn't been made to worker yet
+ // or worker hasn't acknowledged the task yet.
+ PENDING_WORKER_ASSIGN(1, true, RunnerTaskState.PENDING),
- int index;
+ RUNNING(2, false, RunnerTaskState.RUNNING),
+ COMPLETE(3, false, RunnerTaskState.NONE);
- State(int index)
+ private int index;
+ private boolean isPending;
+ private RunnerTaskState runnerTaskState;
Review comment:
changed, for some reason I thought enum state was immutable without even
explicitly doing so.. duh
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356799820
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -313,6 +325,25 @@ private void scheduleCompletedTaskStatusCleanupFromZk()
);
}
+ /**
+ * Must not be used outside of this class and {@link
HttpRemoteTaskRunnerResource}
+ */
+ @SuppressWarnings("GuardedBy") // Read on workersWithUnacknowledgedTask is
safe
+ public Map getWorkersEligibleToRunTasks()
Review comment:
changed, here and in few other similar places
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356800295
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -1432,15 +1533,35 @@ void taskAddedOrUpdated(final TaskAnnouncement
announcement, final WorkerHolder
{
enum State
{
- PENDING(0),
- RUNNING(1),
- COMPLETE(2);
+ // Task has been given to HRTR, but a worker to run this task hasn't
been identified yet.
+ PENDING(0, true, RunnerTaskState.PENDING),
+
+ // A Worker has been identified to run this task, but request to run
task hasn't been made to worker yet
+ // or worker hasn't acknowledged the task yet.
+ PENDING_WORKER_ASSIGN(1, true, RunnerTaskState.PENDING),
- int index;
+ RUNNING(2, false, RunnerTaskState.RUNNING),
+ COMPLETE(3, false, RunnerTaskState.NONE);
- State(int index)
+ private int index;
+ private boolean isPending;
+ private RunnerTaskState runnerTaskState;
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356800260
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -313,6 +325,25 @@ private void scheduleCompletedTaskStatusCleanupFromZk()
);
}
+ /**
+ * Must not be used outside of this class and {@link
HttpRemoteTaskRunnerResource}
+ */
+ @SuppressWarnings("GuardedBy") // Read on workersWithUnacknowledgedTask is
safe
+ public Map getWorkersEligibleToRunTasks()
+ {
+return Maps.transformEntries(
+Maps.filterEntries(
+workers,
+input -> !lazyWorkers.containsKey(input.getKey()) &&
+
!workersWithUnacknowledgedTask.containsKey(input.getKey()) &&
+ !blackListedWorkers.containsKey(input.getKey()) &&
Review comment:
added comment to explain it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] himanshug commented on a change in pull request #8697: HRTR: make pending task execution handling to go through all tasks on not finding worker slots
himanshug commented on a change in pull request #8697: HRTR: make pending task
execution handling to go through all tasks on not finding worker slots
URL: https://github.com/apache/incubator-druid/pull/8697#discussion_r356799820
##
File path:
indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java
##
@@ -313,6 +325,25 @@ private void scheduleCompletedTaskStatusCleanupFromZk()
);
}
+ /**
+ * Must not be used outside of this class and {@link
HttpRemoteTaskRunnerResource}
+ */
+ @SuppressWarnings("GuardedBy") // Read on workersWithUnacknowledgedTask is
safe
+ public Map getWorkersEligibleToRunTasks()
Review comment:
changed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] satybald opened a new pull request #9014: Add extraction possibility from kafka json messages in a druid lookup…
satybald opened a new pull request #9014: Add extraction possibility from kafka json messages in a druid lookup… URL: https://github.com/apache/incubator-druid/pull/9014 Fixes #8097. ### Description PR adds custom Json and Jq extractors for Kafka lookup module. The extractors were adopted from `lookup-cached-global` module. This PR has: - [x] added documentation for new or modified features or behaviors. - [x] added unit tests for custom Json extractors - [x] been self-reviewed. - [x] using the [concurrency checklist](https://github.com/apache/incubator-druid/blob/master/dev/code-review/concurrency.md) - [x] has been tested in production :smiley: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
