[GitHub] [druid] sekingme commented on issue #8983: Enable parameters 'resultAsArray' working in materialized-view-selection with a group by query
sekingme commented on issue #8983: Enable parameters 'resultAsArray' working in materialized-view-selection with a group by query URL: https://github.com/apache/druid/pull/8983#issuecomment-603621641 > @jihoonson Hi, please check. @vogievetsky Execute me, nobody try to review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch 0.18.0 updated: S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured. (#9552) (#9558)
This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch 0.18.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/0.18.0 by this push:
new ef124dc S3 ingestion spec should not uses the default credentials
provider chain when environment value password provider is misconfigured.
(#9552) (#9558)
ef124dc is described below
commit ef124dcace15de7715f70887b6e7a708979549bc
Author: Maytas Monsereenusorn <52679095+mayta...@users.noreply.github.com>
AuthorDate: Tue Mar 24 21:12:16 2020 -0700
S3 ingestion spec should not uses the default credentials provider chain
when environment value password provider is misconfigured. (#9552) (#9558)
* fix s3 optional cred
* S3 ingestion spec uses the default credentials provider chain when
environment value password provider is misconfigured.
* fix failing test
---
.travis.yml| 4 +-
.../druid/data/input/s3/S3InputSourceConfig.java | 4 +-
integration-tests/pom.xml | 6 +
.../clients/OverlordResourceTestClient.java| 35 ++-
.../java/org/apache/druid/tests/TestNGGroup.java | 5 +-
...AbstractAzureInputSourceParallelIndexTest.java} | 2 +-
...> AbstractGcsInputSourceParallelIndexTest.java} | 2 +-
...=> AbstractS3InputSourceParallelIndexTest.java} | 2 +-
.../indexer/ITAzureToAzureParallelIndexTest.java | 2 +-
.../indexer/ITAzureToHdfsParallelIndexTest.java| 2 +-
.../tests/indexer/ITGcsToGcsParallelIndexTest.java | 2 +-
.../indexer/ITGcsToHdfsParallelIndexTest.java | 2 +-
.../indexer/ITS3OverrideCredentialsIndexTest.java | 289 +
.../tests/indexer/ITS3ToHdfsParallelIndexTest.java | 2 +-
.../tests/indexer/ITS3ToS3ParallelIndexTest.java | 2 +-
.../wikipedia_override_credentials_index_task.json | 85 ++
16 files changed, 428 insertions(+), 18 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index dc80ab0..e7582d5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -344,7 +344,7 @@ jobs:
name: "(Compile=openjdk8, Run=openjdk8) other integration test"
jdk: openjdk8
services: *integration_test_services
- env:
TESTNG_GROUPS='-DexcludedGroups=batch-index,perfect-rollup-parallel-batch-index,kafka-index,query,realtime-index,security,s3-deep-storage,gcs-deep-storage,azure-deep-storage,hdfs-deep-storage'
JVM_RUNTIME='-Djvm.runtime=8'
+ env:
TESTNG_GROUPS='-DexcludedGroups=batch-index,perfect-rollup-parallel-batch-index,kafka-index,query,realtime-index,security,s3-deep-storage,gcs-deep-storage,azure-deep-storage,hdfs-deep-storage,s3-ingestion'
JVM_RUNTIME='-Djvm.runtime=8'
script: *run_integration_test
after_failure: *integration_test_diags
# END - Integration tests for Compile with Java 8 and Run with Java 8
@@ -383,7 +383,7 @@ jobs:
- <<: *integration_tests
name: "(Compile=openjdk8, Run=openjdk11) other integration test"
jdk: openjdk8
- env:
TESTNG_GROUPS='-DexcludedGroups=batch-index,perfect-rollup-parallel-batch-index,kafka-index,query,realtime-index,security,s3-deep-storage,gcs-deep-storage,azure-deep-storage,hdfs-deep-storage'
JVM_RUNTIME='-Djvm.runtime=11'
+ env:
TESTNG_GROUPS='-DexcludedGroups=batch-index,perfect-rollup-parallel-batch-index,kafka-index,query,realtime-index,security,s3-deep-storage,gcs-deep-storage,azure-deep-storage,hdfs-deep-storage,s3-ingestion'
JVM_RUNTIME='-Djvm.runtime=11'
# END - Integration tests for Compile with Java 8 and Run with Java 11
- name: "security vulnerabilities"
diff --git
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSourceConfig.java
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSourceConfig.java
index fd27b99..4543449 100644
---
a/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSourceConfig.java
+++
b/extensions-core/s3-extensions/src/main/java/org/apache/druid/data/input/s3/S3InputSourceConfig.java
@@ -67,9 +67,7 @@ public class S3InputSourceConfig
public boolean isCredentialsConfigured()
{
return accessKeyId != null &&
- accessKeyId.getPassword() != null &&
- secretAccessKey != null &&
- secretAccessKey.getPassword() != null;
+ secretAccessKey != null;
}
@Override
diff --git a/integration-tests/pom.xml b/integration-tests/pom.xml
index 7aa13e0..d9cc62d 100644
--- a/integration-tests/pom.xml
+++ b/integration-tests/pom.xml
@@ -38,6 +38,12 @@
${project.parent.version}
+com.amazonaws
+aws-java-sdk-s3
+${aws.sdk.version}
+runtime
+
+
org.apache.druid.extensions
druid-s3-extensions
${project.parent.version}
diff --git
[GitHub] [druid-website-src] hishamitani opened a new pull request #115: Update Added Deep.BI
hishamitani opened a new pull request #115: Update Added Deep.BI URL: https://github.com/apache/druid-website-src/pull/115 Hello, thank you for your consideration! The phrasing I have put in there might have been better in terms of making it clear but the main point is that our platform is built on Druid (on which our UI platform is built as well) as the core with others such as Flink and Cassandra that allow us to augment the way we use Druid. Thanks in advance! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #9558: [Backport] S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured.
clintropolis merged pull request #9558: [Backport] S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured. URL: https://github.com/apache/druid/pull/9558 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart
maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart URL: https://github.com/apache/druid/pull/9560#issuecomment-603635723 What is the motivation for running tests Using A Quickstart Cluster? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s commented on a change in pull request #9559: error on value counter overflow instead of writing sad segments
suneet-s commented on a change in pull request #9559: error on value counter
overflow instead of writing sad segments
URL: https://github.com/apache/druid/pull/9559#discussion_r397581834
##
File path:
processing/src/main/java/org/apache/druid/segment/data/ColumnCapacityExceededException.java
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment.data;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.java.util.common.StringUtils;
+
+public class ColumnCapacityExceededException extends RuntimeException
+{
+ @VisibleForTesting
+ public static String formatMessage(String columnName)
+ {
+return StringUtils.format(
+"Too many values to store for %s column, try reducing
maxRowsPerSegment",
Review comment:
+1 for this operator friendly error message
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] suneet-s opened a new pull request #9560: Instructions to run integration tests against quickstart
suneet-s opened a new pull request #9560: Instructions to run integration tests against quickstart URL: https://github.com/apache/druid/pull/9560 This change makes it easier to run integration tests against a quickstart cluster running on a local development machine. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis opened a new pull request #9559: error on value counter overflow instead of writing sad segments
clintropolis opened a new pull request #9559: error on value counter overflow instead of writing sad segments URL: https://github.com/apache/druid/pull/9559 ### Description This PR fixes an integer overflow issue if too many values are written to a column within a single segment, preferring to fail at segment creation time instead of write a sad segment with negative values. This issue was first noticed with multi-value columns, exploding at query time with an error of the form: ``` org.apache.druid.java.util.common.IAE: Index[-131072] < 0 at org.apache.druid.segment.data.GenericIndexed.checkIndex(GenericIndexed.java:269) at org.apache.druid.segment.data.GenericIndexed.access$300(GenericIndexed.java:79) at org.apache.druid.segment.data.GenericIndexed$3.get(GenericIndexed.java:696) at org.apache.druid.segment.data.CompressedVSizeColumnarIntsSupplier$CompressedVSizeColumnarInts.loadBuffer(CompressedVSizeColumnarIntsSupplier.java:437) at org.apache.druid.segment.data.CompressedVSizeColumnarIntsSupplier$CompressedVSizeColumnarInts.get(CompressedVSizeColumnarIntsSupplier.java:350) at org.apache.druid.segment.data.SliceIndexedInts.get(SliceIndexedInts.java:60) ``` but would also occur for any serializer given more than `Integer.MAX_VALUES` rows as input. To fix, primitive column serializers now check the number of values (row count in most cases, total number of values for the case of multi-value strings) to ensure that it does not extend beyond the values that will be expressed in the column header and won't cause any issues at query time. A new exception, `ColumnCapacityExceededException` has been added which will give an error message that suggests ``` Too many values to store for %s column, try reducing maxRowsPerSegment ``` where `%s` is the column name (which all the serializers now know). I added a bunch of tests to confirm that this works, and also marked them all `@Ignore` because they take forever to run. The same `IAE` error can be replicated by running `V3CompressedVSizeColumnarMultiIntsSerializerTest.testTooManyValues` without the modifications to check that overflow has occurred. I also added a `CompressedDoublesSerdeTest` that copies `CompressedFloatsSerdeTest` since i noticed there wasn't a test for double columns. This PR has: - [ ] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths. - [ ] added integration tests. - [ ] been tested in a test Druid cluster. # Key changed/added classes in this PR * Primitive column serializers and related tests This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 edited a comment on issue #9560: Instructions to run integration tests against quickstart
maytasm3 edited a comment on issue #9560: Instructions to run integration tests against quickstart URL: https://github.com/apache/druid/pull/9560#issuecomment-603635723 What is the motivation for running tests Using A Quickstart Cluster? Is it just mainly for debuggability? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch 0.18.0 created (now 57018ad)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a change to branch 0.18.0 in repository https://gitbox.apache.org/repos/asf/druid.git. at 57018ad change backtick() and fix broken links (#9550) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] stale[bot] commented on issue #7916: does druid-0.14 support PreparedStatement?
stale[bot] commented on issue #7916: does druid-0.14 support PreparedStatement? URL: https://github.com/apache/druid/issues/7916#issuecomment-603082861 This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the d...@druid.apache.org list. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (57018ad -> e97695d)
This is an automated email from the ASF dual-hosted git repository. jihoonson pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from 57018ad change backtick() and fix broken links (#9550) add e97695d fix Hadoop ingestion fails due to error 'JavaScript is disabled' on certain config (#9553) No new revisions were added by this update. Summary of changes: .../indexer/DetermineHashedPartitionsJob.java | 2 +- .../druid/indexer/DeterminePartitionsJob.java | 4 +- .../druid/indexer/HadoopDruidIndexerConfig.java| 20 -- .../apache/druid/indexer/IndexGeneratorJob.java| 6 +- .../java/org/apache/druid/indexer/JobHelper.java | 53 +++ .../org/apache/druid/indexer/JobHelperTest.java| 76 ++ 6 files changed, 120 insertions(+), 41 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson merged pull request #9553: fix Hadoop ingestion fails due to error 'JavaScript is disabled' on certain config
jihoonson merged pull request #9553: fix Hadoop ingestion fails due to error 'JavaScript is disabled' on certain config URL: https://github.com/apache/druid/pull/9553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] sydt2014 removed a comment on issue #9543: broker query error when druid with kerberos
sydt2014 removed a comment on issue #9543: broker query error when druid with kerberos URL: https://github.com/apache/druid/issues/9543#issuecomment-603023970 I can't believe why this code can merge ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito commented on issue #8987: Adding support for autoscaling in GCE
frnidito commented on issue #8987: Adding support for autoscaling in GCE URL: https://github.com/apache/druid/pull/8987#issuecomment-603087781 Ping :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #7916: does druid-0.14 support PreparedStatement?
clintropolis commented on issue #7916: does druid-0.14 support PreparedStatement? URL: https://github.com/apache/druid/issues/7916#issuecomment-603158117 #6974 was merged and will be released in upcoming version 0.18, closing this issue This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis closed issue #7916: does druid-0.14 support PreparedStatement?
clintropolis closed issue #7916: does druid-0.14 support PreparedStatement? URL: https://github.com/apache/druid/issues/7916 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis commented on a change in pull request #8987: Adding support for autoscaling in GCE
clintropolis commented on a change in pull request #8987: Adding support for
autoscaling in GCE
URL: https://github.com/apache/druid/pull/8987#discussion_r397043748
##
File path:
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java
##
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.autoscaling.gce;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.json.jackson2.JacksonFactory;
+import com.google.api.services.compute.Compute;
+import com.google.api.services.compute.ComputeScopes;
+import com.google.api.services.compute.model.Instance;
+import
com.google.api.services.compute.model.InstanceGroupManagersDeleteInstancesRequest;
+import
com.google.api.services.compute.model.InstanceGroupManagersListManagedInstancesResponse;
+import com.google.api.services.compute.model.InstanceList;
+import com.google.api.services.compute.model.ManagedInstance;
+import com.google.api.services.compute.model.NetworkInterface;
+import com.google.api.services.compute.model.Operation;
+import com.google.common.base.Preconditions;
+import com.google.common.net.InetAddresses;
+import
org.apache.curator.shaded.com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScaler;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScalingData;
+import
org.apache.druid.indexing.overlord.autoscaling.SimpleWorkerProvisioningConfig;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * This module permits the autoscaling of the workers in GCE
+ *
+ * General notes:
+ * - The IPs are IPs as in Internet Protocol, and they look like 1.2.3.4
+ * - The IDs are the names of the instances of instances created, they look
like prefix-abcd,
+ * where the prefix is chosen by you and abcd is a suffix assigned by GCE
+ */
+@JsonTypeName("gce")
+public class GceAutoScaler implements AutoScaler
+{
+ private static final EmittingLogger log = new
EmittingLogger(GceAutoScaler.class);
+
+ private final GceEnvironmentConfig envConfig;
+ private final int minNumWorkers;
+ private final int maxNumWorkers;
+ private final SimpleWorkerProvisioningConfig config; // For future use
+
+ private Compute cachedComputeService = null;
+
+ private static final long POLL_INTERVAL_MS = 5 * 1000; // 5 sec
+ private static final int RUNNING_INSTANCES_MAX_RETRIES = 10;
+ private static final int OPERATION_END_MAX_RETRIES = 10;
+
+ @JsonCreator
+ public GceAutoScaler(
+ @JsonProperty("minNumWorkers") int minNumWorkers,
+ @JsonProperty("maxNumWorkers") int maxNumWorkers,
+ @JsonProperty("envConfig") GceEnvironmentConfig envConfig,
+ @JacksonInject SimpleWorkerProvisioningConfig config
+ )
+ {
+Preconditions.checkArgument(minNumWorkers > 0,
+"minNumWorkers must be greater than 0");
+this.minNumWorkers = minNumWorkers;
+Preconditions.checkArgument(maxNumWorkers > 0,
+"maxNumWorkers must be greater than 0");
+Preconditions.checkArgument(maxNumWorkers > minNumWorkers,
+"maxNumWorkers must be greater than
minNumWorkers");
+this.maxNumWorkers = maxNumWorkers;
+this.envConfig = envConfig;
+this.config = config;
+ }
+
+ /**
+ * CAVEAT: this is meant to be used only for testing passing a mock version
of Compute
+ */
+ @VisibleForTesting
+ public GceAutoScaler(
+
[druid] branch master updated (e97695d -> 0ac875a)
This is an automated email from the ASF dual-hosted git repository. cwylie pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/druid.git. from e97695d fix Hadoop ingestion fails due to error 'JavaScript is disabled' on certain config (#9553) add 0ac875a Update docker.md readme to note memory requirements (#9529) No new revisions were added by this update. Summary of changes: docs/tutorials/docker.md | 3 +++ 1 file changed, 3 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] clintropolis merged pull request #9529: Update docker.md readme to note memory requirements
clintropolis merged pull request #9529: Update docker.md readme to note memory requirements URL: https://github.com/apache/druid/pull/9529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] frnidito commented on a change in pull request #8987: Adding support for autoscaling in GCE
frnidito commented on a change in pull request #8987: Adding support for
autoscaling in GCE
URL: https://github.com/apache/druid/pull/8987#discussion_r397057336
##
File path:
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GceAutoScaler.java
##
@@ -0,0 +1,539 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.autoscaling.gce;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.json.jackson2.JacksonFactory;
+import com.google.api.services.compute.Compute;
+import com.google.api.services.compute.ComputeScopes;
+import com.google.api.services.compute.model.Instance;
+import
com.google.api.services.compute.model.InstanceGroupManagersDeleteInstancesRequest;
+import
com.google.api.services.compute.model.InstanceGroupManagersListManagedInstancesResponse;
+import com.google.api.services.compute.model.InstanceList;
+import com.google.api.services.compute.model.ManagedInstance;
+import com.google.api.services.compute.model.NetworkInterface;
+import com.google.api.services.compute.model.Operation;
+import com.google.common.base.Preconditions;
+import com.google.common.net.InetAddresses;
+import
org.apache.curator.shaded.com.google.common.annotations.VisibleForTesting;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScaler;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScalingData;
+import
org.apache.druid.indexing.overlord.autoscaling.SimpleWorkerProvisioningConfig;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * This module permits the autoscaling of the workers in GCE
+ *
+ * General notes:
+ * - The IPs are IPs as in Internet Protocol, and they look like 1.2.3.4
+ * - The IDs are the names of the instances of instances created, they look
like prefix-abcd,
+ * where the prefix is chosen by you and abcd is a suffix assigned by GCE
+ */
+@JsonTypeName("gce")
+public class GceAutoScaler implements AutoScaler
+{
+ private static final EmittingLogger log = new
EmittingLogger(GceAutoScaler.class);
+
+ private final GceEnvironmentConfig envConfig;
+ private final int minNumWorkers;
+ private final int maxNumWorkers;
+ private final SimpleWorkerProvisioningConfig config; // For future use
+
+ private Compute cachedComputeService = null;
+
+ private static final long POLL_INTERVAL_MS = 5 * 1000; // 5 sec
+ private static final int RUNNING_INSTANCES_MAX_RETRIES = 10;
+ private static final int OPERATION_END_MAX_RETRIES = 10;
+
+ @JsonCreator
+ public GceAutoScaler(
+ @JsonProperty("minNumWorkers") int minNumWorkers,
+ @JsonProperty("maxNumWorkers") int maxNumWorkers,
+ @JsonProperty("envConfig") GceEnvironmentConfig envConfig,
+ @JacksonInject SimpleWorkerProvisioningConfig config
+ )
+ {
+Preconditions.checkArgument(minNumWorkers > 0,
+"minNumWorkers must be greater than 0");
+this.minNumWorkers = minNumWorkers;
+Preconditions.checkArgument(maxNumWorkers > 0,
+"maxNumWorkers must be greater than 0");
+Preconditions.checkArgument(maxNumWorkers > minNumWorkers,
+"maxNumWorkers must be greater than
minNumWorkers");
+this.maxNumWorkers = maxNumWorkers;
+this.envConfig = envConfig;
+this.config = config;
+ }
+
+ /**
+ * CAVEAT: this is meant to be used only for testing passing a mock version
of Compute
+ */
+ @VisibleForTesting
+ public GceAutoScaler(
+ int
[GitHub] [druid] clintropolis commented on a change in pull request #8987: Adding support for autoscaling in GCE
clintropolis commented on a change in pull request #8987: Adding support for
autoscaling in GCE
URL: https://github.com/apache/druid/pull/8987#discussion_r397047442
##
File path:
extensions-contrib/gce-extensions/src/main/java/org/apache/druid/indexing/overlord/autoscaling/gce/GCEAutoScaler.java
##
@@ -0,0 +1,496 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.indexing.overlord.autoscaling.gce;
+
+import com.fasterxml.jackson.annotation.JacksonInject;
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
+import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
+import com.google.api.client.http.HttpTransport;
+import com.google.api.client.json.JsonFactory;
+import com.google.api.client.json.jackson2.JacksonFactory;
+import com.google.api.services.compute.Compute;
+import com.google.api.services.compute.ComputeScopes;
+import com.google.api.services.compute.model.Instance;
+import
com.google.api.services.compute.model.InstanceGroupManagersDeleteInstancesRequest;
+import
com.google.api.services.compute.model.InstanceGroupManagersListManagedInstancesResponse;
+import com.google.api.services.compute.model.InstanceList;
+import com.google.api.services.compute.model.ManagedInstance;
+import com.google.api.services.compute.model.Operation;
+import com.google.common.net.InetAddresses;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScaler;
+import org.apache.druid.indexing.overlord.autoscaling.AutoScalingData;
+import
org.apache.druid.indexing.overlord.autoscaling.SimpleWorkerProvisioningConfig;
+import org.apache.druid.java.util.emitter.EmittingLogger;
+
+import java.io.IOException;
+import java.security.GeneralSecurityException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+
+/**
+ * This module permits the autoscaling of the workers in GCE
+ *
+ * General notes:
+ * - The IPs are IPs as in Internet Protocol, and they look like 1.2.3.4
+ * - The IDs are the names of the instances of instances created, they look
like prefix-abcd,
+ * where the prefix is chosen by you and abcd is a suffix assigned by GCE
+ */
+@JsonTypeName("gce")
+public class GCEAutoScaler implements AutoScaler
+{
+ private static final EmittingLogger log = new
EmittingLogger(GCEAutoScaler.class);
+
+ private final GCEEnvironmentConfig envConfig;
+ private final int minNumWorkers;
+ private final int maxNumWorkers;
+ private final SimpleWorkerProvisioningConfig config; // For future use
+
+ private Compute cachedComputeService = null;
+
+ @JsonCreator
+ public GCEAutoScaler(
+ @JsonProperty("minNumWorkers") int minNumWorkers,
+ @JsonProperty("maxNumWorkers") int maxNumWorkers,
+ @JsonProperty("envConfig") GCEEnvironmentConfig envConfig,
+ @JacksonInject SimpleWorkerProvisioningConfig config
+ )
+ {
+this.minNumWorkers = minNumWorkers;
+this.maxNumWorkers = maxNumWorkers;
+this.envConfig = envConfig;
+this.config = config;
+ }
+
+ /**
+ * CAVEAT: this is meant to be used only for testing passing a mock version
of Compute
+ */
+ public GCEAutoScaler(
+ int minNumWorkers,
+ int maxNumWorkers,
+ GCEEnvironmentConfig envConfig,
+ SimpleWorkerProvisioningConfig config,
+ Compute compute
+ )
+ {
+this(minNumWorkers, maxNumWorkers, envConfig, config);
+this.cachedComputeService = compute;
+ }
+
+ @Override
+ @JsonProperty
+ public int getMinNumWorkers()
+ {
+return minNumWorkers;
+ }
+
+ @Override
+ @JsonProperty
+ public int getMaxNumWorkers()
+ {
+return maxNumWorkers;
+ }
+
+ @Override
+ @JsonProperty
+ public GCEEnvironmentConfig getEnvConfig()
+ {
+return envConfig;
+ }
+
+ private synchronized Compute createComputeService()
+ throws IOException, GeneralSecurityException, InterruptedException
+ {
+final int max_retries = 5;
+final long retries_interval = 5 * 1000; // 5 secs.
+
+int retries = 0;
+while
[GitHub] [druid] vogievetsky merged pull request #9549: Add view values to lookup actions menu
vogievetsky merged pull request #9549: Add view values to lookup actions menu URL: https://github.com/apache/druid/pull/9549 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] vogievetsky commented on issue #9549: Add view values to lookup actions menu
vogievetsky commented on issue #9549: Add view values to lookup actions menu URL: https://github.com/apache/druid/pull/9549#issuecomment-603368340 Looks great, thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (0ac875a -> e1b201c)
This is an automated email from the ASF dual-hosted git repository.
vogievetsky pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.
from 0ac875a Update docker.md readme to note memory requirements (#9529)
add e1b201c Add view values to lookup actions menu (#9549)
No new revisions were added by this update.
Summary of changes:
.../lookup-values-table.spec.tsx.snap} | 2 +-
.../lookup-values-table.scss} | 2 +-
.../lookup-values-table.spec.tsx} | 4 +-
.../lookup-values-table.tsx} | 56 ++
.../lookup-table-action-dialog.spec.tsx.snap} | 8 ++--
.../lookup-table-action-dialog.spec.tsx} | 13 +++--
.../lookup-table-action-dialog.tsx}| 30 ++--
.../src/views/lookups-view/lookups-view.tsx| 26 +-
8 files changed, 79 insertions(+), 62 deletions(-)
copy
web-console/src/components/{datasource-columns-table/__snapshots__/datasource-columns-table.spec.tsx.snap
=> lookup-values-table/__snapshots__/lookup-values-table.spec.tsx.snap} (97%)
copy
web-console/src/components/{datasource-columns-table/datasource-columns-table.scss
=> lookup-values-table/lookup-values-table.scss} (97%)
copy web-console/src/components/{show-json/show-json.spec.tsx =>
lookup-values-table/lookup-values-table.spec.tsx} (87%)
copy
web-console/src/components/{datasource-columns-table/datasource-columns-table.tsx
=> lookup-values-table/lookup-values-table.tsx} (58%)
copy
web-console/src/dialogs/{datasource-table-action-dialog/__snapshots__/datasource-table-action-dialog.spec.tsx.snap
=>
lookup-table-action-dialog/__snapshots__/lookup-table-action-dialog.spec.tsx.snap}
(97%)
copy
web-console/src/dialogs/{segments-table-action-dialog/segment-table-action-dialog.spec.tsx
=> lookup-table-action-dialog/lookup-table-action-dialog.spec.tsx} (79%)
copy
web-console/src/dialogs/{datasource-table-action-dialog/datasource-table-action-dialog.tsx
=> lookup-table-action-dialog/lookup-table-action-dialog.tsx} (63%)
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9507: optionally disable all of hardcoded zookeeper use
himanshug commented on a change in pull request #9507: optionally disable all
of hardcoded zookeeper use
URL: https://github.com/apache/druid/pull/9507#discussion_r397314006
##
File path:
server/src/main/java/org/apache/druid/server/http/HistoricalResource.java
##
@@ -50,14 +50,14 @@ public HistoricalResource(
@Produces(MediaType.APPLICATION_JSON)
public Response getLoadStatus()
{
-return Response.ok(ImmutableMap.of("cacheInitialized",
coordinator.isStarted())).build();
+return Response.ok(ImmutableMap.of("cacheInitialized",
segmentLoadDropHandler.isStarted())).build();
}
@GET
@Path("/readiness")
public Response getReadiness()
{
-if (coordinator.isStarted()) {
+if (segmentLoadDropHandler.isStarted()) {
Review comment:
primary thing checked by this endpoint is that historical startup
sequence has finished loading/announcing segments that it already had on
disk which is often a time consuming activity. that is still ensured by
`segmentLoadDropHandler.isStarted()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9507: optionally disable all of hardcoded zookeeper use
himanshug commented on a change in pull request #9507: optionally disable all
of hardcoded zookeeper use
URL: https://github.com/apache/druid/pull/9507#discussion_r397315538
##
File path: server/src/main/java/org/apache/druid/curator/CuratorModule.java
##
@@ -63,15 +63,20 @@
@Override
public void configure(Binder binder)
{
+JsonConfigProvider.bind(binder, CURATOR_CONFIG_PREFIX,
ZkEnablementConfig.class);
JsonConfigProvider.bind(binder, CURATOR_CONFIG_PREFIX,
CuratorConfig.class);
JsonConfigProvider.bind(binder, EXHIBITOR_CONFIG_PREFIX,
ExhibitorConfig.class);
}
@Provides
@LazySingleton
@SuppressForbidden(reason = "System#err")
- public CuratorFramework makeCurator(CuratorConfig config, EnsembleProvider
ensembleProvider, Lifecycle lifecycle)
+ public CuratorFramework makeCurator(ZkEnablementConfig zkEnablementConfig,
CuratorConfig config, EnsembleProvider ensembleProvider, Lifecycle lifecycle)
Review comment:
Main reason for failing loudly here is so that if I forgot to disable zk in
some code path, then this method would immediately fail on node start with
guice injection errors leading to quick discovery of exactly what is missed.
this helped me catch quite a few places that I missed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9507: optionally disable all of hardcoded zookeeper use
himanshug commented on a change in pull request #9507: optionally disable all
of hardcoded zookeeper use
URL: https://github.com/apache/druid/pull/9507#discussion_r397316647
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinator.java
##
@@ -443,7 +455,7 @@ public void moveSegment(
() -> {
try {
if
(serverInventoryView.isSegmentLoadedByServer(toServer.getName(), segment) &&
-curator.checkExists().forPath(toLoadQueueSegPath) == null
&&
+(curator == null ||
curator.checkExists().forPath(toLoadQueueSegPath) == null) &&
Review comment:
yeah, I hope zk based segment management will just go away and this will go
as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9507: optionally disable all of hardcoded zookeeper use
himanshug commented on a change in pull request #9507: optionally disable all
of hardcoded zookeeper use
URL: https://github.com/apache/druid/pull/9507#discussion_r397317523
##
File path:
server/src/main/java/org/apache/druid/server/coordinator/LoadQueueTaskMaster.java
##
@@ -67,7 +68,7 @@ public LoadQueuePeon giveMePeon(ImmutableDruidServer server)
return new HttpLoadQueuePeon(server.getURL(), jsonMapper, httpClient,
config, peonExec, callbackExec);
} else {
return new CuratorLoadQueuePeon(
- curator,
+ curatorFrameworkProvider.get(),
Review comment:
could be, but you know my preference by now :)
I wish people would migrate to using `HTTP` segment management and that
would remain the only way and this would be deleted.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on a change in pull request #9507: optionally disable all of hardcoded zookeeper use
himanshug commented on a change in pull request #9507: optionally disable all
of hardcoded zookeeper use
URL: https://github.com/apache/druid/pull/9507#discussion_r397320944
##
File path:
server/src/main/java/org/apache/druid/server/coordination/BatchDataSegmentAnnouncer.java
##
@@ -99,13 +107,28 @@ public BatchDataSegmentAnnouncer(
return rv;
};
-if (this.config.isSkipSegmentAnnouncementOnZk()) {
+isSkipSegmentAnnouncementOnZk = !zkEnablementConfig.isEnabled() ||
config.isSkipSegmentAnnouncementOnZk();
Review comment:
this is also calling methods in
`private final ChangeRequestHistory changes = new
ChangeRequestHistory<>();`
which is used by the http endpoint for segment sync.
However, yeah it takes more work to announce stuff in zk so it looks like
that this class is primarily doing that whereas it is actually supporting both.
If/When we are able to delete zk based segment management this class would
shrink significantly.
Also, unrelated, it could possibly be refactored to separate two things it
is doing so that things are more clear.
However, you know my preference, just delete zk stuff :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on issue #9507: optionally disable all of hardcoded zookeeper use
himanshug commented on issue #9507: optionally disable all of hardcoded zookeeper use URL: https://github.com/apache/druid/pull/9507#issuecomment-603378061 @clintropolis thanks for clarification but I took the comments as in coming from a curious reviewer. In fact, with written communication, I always try to imagine a happy person speaking saying things. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug edited a comment on issue #9507: optionally disable all of hardcoded zookeeper use
himanshug edited a comment on issue #9507: optionally disable all of hardcoded zookeeper use URL: https://github.com/apache/druid/pull/9507#issuecomment-603378061 @clintropolis thanks for clarification but I took the comments as in coming from a curious reviewer. In fact, with written communication, I always try to imagine a happy person speaking things. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] xvrl commented on issue #9546: support schemaless ingestion for transformed dimensions
xvrl commented on issue #9546: support schemaless ingestion for transformed dimensions URL: https://github.com/apache/druid/pull/9546#issuecomment-603379060 @clintropolis agree there are some quirks, it's a bit unfortunate that dimensionExclusions don't get applied after transforms, otherwise we could use that as a workaround. One option would be to pass in an additional field to the transform spec to indicate which field should be added as dimensions. The behavior might still be odd since those new dimensions would still be added even if they are not part of the schema when one is explicitly defined. Maybe we can add this as an experimental/undocumented feature for now and see if we find a better solution long-term. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid-website-src] branch master updated: Update events.yml
This is an automated email from the ASF dual-hosted git repository. fjy pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid-website-src.git The following commit(s) were added to refs/heads/master by this push: new 2790662 Update events.yml new de37960 Merge pull request #114 from rachelpedreschi/patch-9 2790662 is described below commit 2790662feafe0a5e8717bab990b88ccee411c661 Author: Rachel Pedreschi <53063158+rachelpedres...@users.noreply.github.com> AuthorDate: Tue Mar 24 10:18:30 2020 -0700 Update events.yml --- _data/events.yml | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/_data/events.yml b/_data/events.yml index 5715c09..f3bfffe 100644 --- a/_data/events.yml +++ b/_data/events.yml @@ -1,9 +1,9 @@ -- date: 2020-03-04 - name: "Analytics at speed - the rise of the Druid" - info: "Castlereagh Boutique Hotel, 169 Castlereagh St, Sydney" - link: https://www.meetup.com/Big-Data-Analytics/events/268442945/ +- date: 2020-03-24 + name: "Virtual Meetup- AGA (*Ask Gian Anything*)" + info: "Hosted by the Bay Area Apache Druid Meetup Group" + link: https://www.meetup.com/druidio/events/269554006/ - date: 2020-04-13 - name: "Druid Summit" - info: "San Francisco Airport Marriott Waterfront, 1800 Old Bayshore Highway, Burlingame, CA" + name: "Virtual Druid Summit" + info: "The comfort of your own laptop" link: https://druidsummit.org/ - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] fjy merged pull request #114: Update events.yml
fjy merged pull request #114: Update events.yml URL: https://github.com/apache/druid-website-src/pull/114 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid-website-src] rachelpedreschi opened a new pull request #114: Update events.yml
rachelpedreschi opened a new pull request #114: Update events.yml URL: https://github.com/apache/druid-website-src/pull/114 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib
himanshug commented on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-603393558 @jon-wei thanks for the review. Also, I have been researching a bit to see if there is a programmatic way to do the `user/resource_owner <--> oauth_server/identity_provider/okta_server` integration to get right tokens , however I couldn't find it. This extension currently implements https://aaronparecki.com/oauth-2-simplified/#web-server-apps flow so that user's access to router web console could be protected using their company's existing enterprise authentication system if it is OAuth and OpenID Connect based (..most company's use Okta for that). However, I can't see even a Okta specific way of automating that part of user auth. That said, I am not an OAuth expert(it is non-trivial with multiple versions/revisions/extensions) and could be missing something. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9552: S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured.
jihoonson commented on issue #9552: S3 ingestion spec should not uses the
default credentials provider chain when environment value password provider is
misconfigured.
URL: https://github.com/apache/druid/pull/9552#issuecomment-603393282
Please fix the checkstyle error.
```
[ERROR]
/home/travis/build/apache/druid/integration-tests/src/test/java/org/apache/druid/tests/indexer/ITS3OverrideCredentialsIndexTest.java:36:
Using a static member import should be avoided -
org.apache.directory.api.util.IOUtils.closeQuietly. [AvoidStaticImport]
[ERROR]
/home/travis/build/apache/druid/integration-tests/src/test/java/org/apache/druid/tests/indexer/ITS3OverrideCredentialsIndexTest.java:194:5:
'}' at column 5 should be alone on a line. [RightCurly]
[ERROR]
/home/travis/build/apache/druid/integration-tests/src/test/java/org/apache/druid/tests/indexer/ITS3OverrideCredentialsIndexTest.java:271:5:
'}' at column 5 should be alone on a line. [RightCurly]
[ERROR]
/home/travis/build/apache/druid/integration-tests/src/test/java/org/apache/druid/tests/indexer/ITS3OverrideCredentialsIndexTest.java:277:50:
'{' at column 50 should be on a new line. [LeftCurly]
[ERROR]
/home/travis/build/apache/druid/integration-tests/src/test/java/org/apache/druid/tests/indexer/ITS3OverrideCredentialsIndexTest.java:282:5:
'}' at column 5 should be alone on a line. [RightCurly]
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] himanshug edited a comment on issue #8992: druid extension for OpenID Connect auth using pac4j lib
himanshug edited a comment on issue #8992: druid extension for OpenID Connect auth using pac4j lib URL: https://github.com/apache/druid/pull/8992#issuecomment-603393558 @jon-wei thanks for the review. Also, I have been researching a bit to see if there is a programmatic way to do the `user/resource_owner <--> oauth_server/identity_provider/okta_server` integration to get right tokens , however I couldn't find it. This extension currently implements https://aaronparecki.com/oauth-2-simplified/#web-server-apps flow so that user's access to router web console could be protected using their company's existing enterprise authentication system if it is OAuth and OpenID Connect based (..most companyies use Okta for that I think). However, I can't see even a Okta specific way of automating that part of user auth. That said, I am not an OAuth expert(it is non-trivial with multiple versions/revisions/extensions) and could be missing something. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 commented on issue #9552: S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured.
maytasm3 commented on issue #9552: S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured. URL: https://github.com/apache/druid/pull/9552#issuecomment-603444116 @clintropolis @jihoonson Fixed CI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9552: S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured.
jihoonson commented on issue #9552: S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured. URL: https://github.com/apache/druid/pull/9552#issuecomment-603530636 The latest travis build has passed: https://travis-ci.org/github/apache/druid/builds/666481903. Merging this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[druid] branch master updated (e1b201c -> 3f52194)
This is an automated email from the ASF dual-hosted git repository.
jihoonson pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git.
from e1b201c Add view values to lookup actions menu (#9549)
add 3f52194 S3 ingestion spec should not uses the default credentials
provider chain when environment value password provider is misconfigured.
(#9552)
No new revisions were added by this update.
Summary of changes:
.travis.yml| 4 +-
.../druid/data/input/s3/S3InputSourceConfig.java | 4 +-
integration-tests/pom.xml | 6 +
.../clients/OverlordResourceTestClient.java| 35 ++-
.../java/org/apache/druid/tests/TestNGGroup.java | 5 +-
...AbstractAzureInputSourceParallelIndexTest.java} | 2 +-
...> AbstractGcsInputSourceParallelIndexTest.java} | 2 +-
...=> AbstractS3InputSourceParallelIndexTest.java} | 2 +-
.../indexer/ITAzureToAzureParallelIndexTest.java | 2 +-
.../indexer/ITAzureToHdfsParallelIndexTest.java| 2 +-
.../tests/indexer/ITGcsToGcsParallelIndexTest.java | 2 +-
.../indexer/ITGcsToHdfsParallelIndexTest.java | 2 +-
.../indexer/ITS3OverrideCredentialsIndexTest.java | 289 +
.../tests/indexer/ITS3ToHdfsParallelIndexTest.java | 2 +-
.../tests/indexer/ITS3ToS3ParallelIndexTest.java | 2 +-
...wikipedia_override_credentials_index_task.json} | 4 +
16 files changed, 347 insertions(+), 18 deletions(-)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{AbstractAzureInputSourceSimpleIndexTest.java
=> AbstractAzureInputSourceParallelIndexTest.java} (97%)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{AbstractGcsInputSourceSimpleIndexTest.java
=> AbstractGcsInputSourceParallelIndexTest.java} (97%)
rename
integration-tests/src/test/java/org/apache/druid/tests/indexer/{AbstractS3InputSourceSimpleIndexTest.java
=> AbstractS3InputSourceParallelIndexTest.java} (97%)
create mode 100644
integration-tests/src/test/java/org/apache/druid/tests/indexer/ITS3OverrideCredentialsIndexTest.java
copy
integration-tests/src/test/resources/indexer/{wikipedia_cloud_simple_index_task.json
=> wikipedia_override_credentials_index_task.json} (92%)
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] maytasm3 opened a new pull request #9558: [Backport] S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured
maytasm3 opened a new pull request #9558: [Backport] S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured. URL: https://github.com/apache/druid/pull/9558 [Backport] S3 ingestion spec should not uses the default credentials provider chain when environment value password provider is misconfigured. ### Description This PR fixes the bug that S3 ingestion spec uses the default credentials provider chain when environment value password provider is misconfigured. The new and correct behavior is that If environment value password provider is misconfigured or any other cases that resulted in the s3 access key and/or s3 secret key from the PasswordProvider being null, the ingestion will failed. The task logs will then surface error message "IllegalArgumentException: Access key cannot be null" to let the user that Access key or secret key was null. Note that ingestion will also fail if only one PasswordProvider is configured (PasswordProvider given for Access key but not Secret key OR PasswordProvider given for Secret key but not Access key). This PR has: - [x] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [ ] added unit tests or modified existing tests to cover new code paths. - [x] added integration tests. - [x] been tested in a test Druid cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
