gianm commented on a change in pull request #8078: Upgrade Kafka library for
kafka-lookup module
URL: https://github.com/apache/incubator-druid/pull/8078#discussion_r304700780
##########
File path:
extensions-core/kafka-extraction-namespace/src/main/java/org/apache/druid/query/lookup/KafkaLookupExtractorFactory.java
##########
@@ -413,4 +353,47 @@ AtomicLong getDoubleEventCount()
{
return future;
}
+
+ private void verifyKafkaProperties()
+ {
+ if (kafkaProperties.containsKey(ConsumerConfig.GROUP_ID_CONFIG)) {
+ throw new IAE(
+ "Cannot set kafka property [group.id]. Property is randomly
generated for you. Found [%s]",
+ kafkaProperties.get(ConsumerConfig.GROUP_ID_CONFIG)
+ );
+ }
+ if (kafkaProperties.containsKey(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG)) {
+ throw new IAE(
+ "Cannot set kafka property [auto.offset.reset]. Property will be
forced to [smallest]. Found [%s]",
+ kafkaProperties.get(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG)
+ );
+ }
+ Preconditions.checkNotNull(
+ kafkaProperties.get(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG),
+ "bootstrap.servers required property"
+ );
+ }
+
+ // Overridden in tests
+ Consumer<String, String> getConsumer()
+ {
+ // Workaround for Kafka String Serializer could not be found
+ // Adopted from - https://stackoverflow.com/a/54118010/2586315
Review comment:
FYI, this comment has sparked a legal discussion:
https://issues.apache.org/jira/browse/LEGAL-467
Due to the fact that StackOverflow uses a Category X (forbidden) license:
https://www.apache.org/legal/resolved.html#stackoverflow
My feeling is it should be fine to use this code, since there isn't any real
creativity in the original SO answer. But I've raised it with ASF legal for
their opinion.
By the way, we do have code for adjusting Kafka Consumer classloaders in
KafkaIndexTask (see `newConsumer`) that is very similar to this code, but
different, and which was presumably not taken from the same SO thread. A path
forward that side-steps the legal analysis of whether or not the SO snippet is
copyrightable might be factoring that existing Druid code out into a helper
method, and calling it here. Or adapting the code here from that other Druid
code, rather than from SO.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
