[GitHub] [druid] liuxiaohui1221 opened a new pull request #10861: coordinator compact filter locked interval before submitted
liuxiaohui1221 opened a new pull request #10861:
URL: https://github.com/apache/druid/pull/10861
Fixes #.
Coordinator submit compaction task sometimes failed,because timechunkLock is
locked by other task(like kafka indexing task).
### Description
The purpose of this PR is to reduce the failure rate of CompactTask and
increase the success rate of CompactTask when useing timechunk lock task.
Fixed the bug ...
Renamed the class ...
Added a forbidden-apis entry ...
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency
checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md)
(Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in
[licenses.yaml](https://github.com/apache/druid/blob/master/licenses.yaml)
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
# Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] kaka11chen commented on pull request #10844: Fix a kind of nested groupBy sql will caused the infinite loop by removing AggregateRemoveRule.
kaka11chen commented on pull request #10844: URL: https://github.com/apache/druid/pull/10844#issuecomment-774608594 @asdf2014 Hi，Benedict，could you please review it or is there any better resolution? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] bananaaggle commented on a change in pull request #10839: Add protobuf schema registry
bananaaggle commented on a change in pull request #10839: URL: https://github.com/apache/druid/pull/10839#discussion_r571522298 ## File path: distribution/bin/check-licenses.py ## @@ -225,6 +225,9 @@ def build_compatible_license_names(): compatible_licenses['Apache License v2.0'] = 'Apache License version 2.0' compatible_licenses['Apache License, version 2.0'] = 'Apache License version 2.0' compatible_licenses['Apache 2.0 License'] = 'Apache License version 2.0' +compatible_licenses['Apache License, 2.0'] = 'Apache License version 2.0' +compatible_licenses['Confluent Community License'] = 'Confluent Community License' Review comment: https://user-images.githubusercontent.com/24642075/107135019-34cc0800-6932-11eb-984d-acdf4be844a0.png;> This is dependency report generated by druid. I think it means kafka-protobuf-provider is licensed by Confluent Community License and Apache License. I change this python script because it can not work if this map lose key 'Confluent Community License'. https://user-images.githubusercontent.com/24642075/107135162-4366ef00-6933-11eb-87c2-a854a57f5630.png;> https://user-images.githubusercontent.com/24642075/107135165-46fa7600-6933-11eb-8e22-c403ef3edcb5.png;> In MVN Repository, it marked as Apache license. So can I use it in Druid packages? I'm not very sure about that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] bananaaggle commented on a change in pull request #10839: Add protobuf schema registry
bananaaggle commented on a change in pull request #10839:
URL: https://github.com/apache/druid/pull/10839#discussion_r571520136
##
File path:
extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/SchemaRegistryBasedProtobufBytesDecoder.java
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input.protobuf;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
+import io.confluent.kafka.schemaregistry.protobuf.ProtobufSchema;
+import io.confluent.kafka.schemaregistry.protobuf.ProtobufSchemaProvider;
+import org.apache.druid.java.util.common.logger.Logger;
+import org.apache.druid.java.util.common.parsers.ParseException;
+
+import java.nio.ByteBuffer;
+import java.util.Collections;
+
+public class SchemaRegistryBasedProtobufBytesDecoder implements
ProtobufBytesDecoder
+{
+
+ private static final Logger LOGGER = new
Logger(SchemaRegistryBasedProtobufBytesDecoder.class);
+
+ private final SchemaRegistryClient registry;
+ private int identityMapCapacity;
+
+ @JsonCreator
+ public SchemaRegistryBasedProtobufBytesDecoder(
+ @JsonProperty("url") String url,
+ @JsonProperty("capacity") Integer capacity
+ )
+ {
+this.identityMapCapacity = capacity == null ? Integer.MAX_VALUE : capacity;
+registry = new CachedSchemaRegistryClient(Collections.singletonList(url),
identityMapCapacity, Collections.singletonList(new ProtobufSchemaProvider()),
null);
+ }
+
+ @VisibleForTesting
+ int getIdentityMapCapacity()
+ {
+return this.identityMapCapacity;
+ }
+
+ @VisibleForTesting
+ SchemaRegistryBasedProtobufBytesDecoder(SchemaRegistryClient registry)
+ {
+this.registry = registry;
+ }
+
+ @Override
+ public DynamicMessage parse(ByteBuffer bytes)
+ {
+try {
+ bytes.get(); // ignore first \0 byte
+ int id = bytes.getInt(); // extract schema registry id
+ bytes.get(); // ignore \0 byte before PB message
+ int length = bytes.limit() - 2 - 4;
+ ProtobufSchema schema = (ProtobufSchema) registry.getSchemaById(id);
+ Descriptors.Descriptor descriptor = schema.toDescriptor();
+ byte[] rawMessage = new byte[length];
+ bytes.get(rawMessage, 0, length);
+ DynamicMessage message = DynamicMessage.parseFrom(descriptor,
rawMessage);
+ return message;
+}
+catch (Exception e) {
Review comment:
I'll split it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] bananaaggle commented on a change in pull request #10839: Add protobuf schema registry
bananaaggle commented on a change in pull request #10839:
URL: https://github.com/apache/druid/pull/10839#discussion_r571512016
##
File path:
extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/FileBasedProtobufBytesDecoder.java
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.data.input.protobuf;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.protobuf.ByteString;
+import com.google.protobuf.Descriptors;
+import com.google.protobuf.DynamicMessage;
+import io.confluent.kafka.schemaregistry.protobuf.dynamic.DynamicSchema;
+import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.parsers.ParseException;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.util.Set;
+
+public class FileBasedProtobufBytesDecoder implements ProtobufBytesDecoder
+{
+ private final String descriptorFilePath;
+ private final String protoMessageType;
+ private Descriptors.Descriptor descriptor;
+
+
+ @JsonCreator
+ public FileBasedProtobufBytesDecoder(
+ @JsonProperty("descriptor") String descriptorFilePath,
+ @JsonProperty("protoMessageType") String protoMessageType
+ )
+ {
+this.descriptorFilePath = descriptorFilePath;
+this.protoMessageType = protoMessageType;
+initDescriptor();
+ }
+
+ @VisibleForTesting
+ void initDescriptor()
+ {
+if (this.descriptor == null) {
+ this.descriptor = getDescriptor(descriptorFilePath);
+}
+ }
+
+ @SuppressWarnings("checkstyle:RightCurly")
Review comment:
I'll delete it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] bananaaggle commented on a change in pull request #10839: Add protobuf schema registry
bananaaggle commented on a change in pull request #10839:
URL: https://github.com/apache/druid/pull/10839#discussion_r571512005
##
File path:
extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/ProtobufInputRowParser.java
##
@@ -37,39 +32,32 @@
import org.apache.druid.data.input.MapBasedInputRow;
import org.apache.druid.data.input.impl.JSONParseSpec;
import org.apache.druid.data.input.impl.ParseSpec;
-import org.apache.druid.java.util.common.StringUtils;
+import org.apache.druid.java.util.common.logger.Logger;
import org.apache.druid.java.util.common.parsers.ParseException;
import org.apache.druid.java.util.common.parsers.Parser;
import org.apache.druid.utils.CollectionUtils;
-import java.io.IOException;
-import java.io.InputStream;
-import java.net.MalformedURLException;
-import java.net.URL;
import java.nio.ByteBuffer;
import java.util.List;
import java.util.Map;
-import java.util.Set;
public class ProtobufInputRowParser implements ByteBufferInputRowParser
{
+ private static final Logger LOG = new Logger(ByteBufferInputRowParser.class);
+
private final ParseSpec parseSpec;
- private final String descriptorFilePath;
- private final String protoMessageType;
- private Descriptor descriptor;
+ private final ProtobufBytesDecoder protobufBytesDecoder;
private Parser parser;
private final List dimensions;
@JsonCreator
public ProtobufInputRowParser(
@JsonProperty("parseSpec") ParseSpec parseSpec,
- @JsonProperty("descriptor") String descriptorFilePath,
- @JsonProperty("protoMessageType") String protoMessageType
+ @JsonProperty("protoBytesDecoder") ProtobufBytesDecoder
protobufBytesDecoder
Review comment:
Good idea, I'll implement it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
jihoonson commented on issue #10752: URL: https://github.com/apache/druid/issues/10752#issuecomment-774529269 Hi @Gabriel39, the kubernetes extension is added in 0.21.0 and not available in the latest release (0.20.1) yet. The doc will be updated once 0.21.0 is released. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] Gabriel39 edited a comment on issue #10752: [Draft] 0.21.0 Release Notes
Gabriel39 edited a comment on issue #10752: URL: https://github.com/apache/druid/issues/10752#issuecomment-774485685 Hi @jihoonson this link https://druid.apache.org/docs/latest/development/extensions-core/kubernetes.html seems like outdated with 404 error. Is there a new site to refer about k8s extension? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [druid] Gabriel39 commented on issue #10752: [Draft] 0.21.0 Release Notes
Gabriel39 commented on issue #10752: URL: https://github.com/apache/druid/issues/10752#issuecomment-774485685 Hi @jihoonson this link https://druid.apache.org/docs/latest/development/extensions-core/kubernetes.html seems like outdated. Is there a new site to refer about k8s extension? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
