Thelin90 opened a new issue #11315: URL: https://github.com/apache/druid/issues/11315
Hello. I am currently working on `Apache Druid` on `K8S`. I want to be able to do some `lookups` towards a `MYSQL` we host in `RDS`. Some more context to the problem: Currently in druid the `classpath` is set to (`And I can verify this inside the pod`): ```bash /tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/master/coordinator-overlord:lib/*: ``` However since version 0.20.02: https://newreleases.io/project/github/apache/druid/release/druid-0.20.2 They also require you to set some parameters for the `jdbc`. They also mention here: * https://druid.apache.org/docs/latest/development/extensions-core/lookups-cached-global.html * https://druid.apache.org/docs/latest/development/extensions-core/druid-lookups.html That you need the `jar` to be put in: * extensions/druid-lookups-cached-single * extensions/druid-lookups-cached-global I can see in the logs, the jar is picked up via the logs: ```bash 2021-05-28T10:42:36,636 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-histogram], jars: druid-histogram-0.21.0.jar 2021-05-28T10:42:36,651 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.21.0.jar 2021-05-28T10:42:36,653 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-global], jars: mapdb-1.0.8.jar, postgresql-42.2.14.jar, druid-lookups-cached-global-0.21.0.jar, mysql-connector-java-5.1.48.jar 2021-05-28T10:42:36,746 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-0.21.0.jar, postgresql-42.2.14.jar 2021-05-28T10:42:36,748 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-s3-extensions], jars: druid-s3-extensions-0.21.0.jar 2021-05-28T10:42:36,753 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: snappy-java-1.1.7.7.jar, zstd-jni-1.3.3-1.jar, kafka-clients-2.7.0.jar, druid-kafka-indexing-service-0.21.0.jar, lz4-java-1.7.1.jar 2021-05-28T10:42:36,757 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-basic-security], jars: druid-basic-security-0.21.0.jar 2021-05-28T10:42:36,760 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-single], jars: antlr-2.7.7.jar, mapdb-1.0.8.jar, druid-lookups-cached-single-0.21.0.jar, postgresql-42.2.14.jar, stringtemplate-3.2.jar, mysql-connector-java-5.1.48.jar 2021-05-28T10:42:37,035 INFO [main] org.apache.druid.cli.CliCoordinator - Coordinator is configured to act as Overlord as well (druid.coordinator.asOverlord.enabled = true). 2021-05-28T10:42:37,563 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-histogram], jars: druid-histogram-0.21.0.jar 2021-05-28T10:42:37,575 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-datasketches], jars: commons-math3-3.6.1.jar, druid-datasketches-0.21.0.jar 2021-05-28T10:42:37,596 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-global], jars: mapdb-1.0.8.jar, postgresql-42.2.14.jar, druid-lookups-cached-global-0.21.0.jar, mysql-connector-java-5.1.48.jar 2021-05-28T10:42:37,602 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [postgresql-metadata-storage], jars: postgresql-metadata-storage-0.21.0.jar, postgresql-42.2.14.jar 2021-05-28T10:42:37,607 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-s3-extensions], jars: druid-s3-extensions-0.21.0.jar 2021-05-28T10:42:37,615 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-kafka-indexing-service], jars: snappy-java-1.1.7.7.jar, zstd-jni-1.3.3-1.jar, kafka-clients-2.7.0.jar, druid-kafka-indexing-service-0.21.0.jar, lz4-java-1.7.1.jar 2021-05-28T10:42:37,625 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-basic-security], jars: druid-basic-security-0.21.0.jar 2021-05-28T10:42:37,642 INFO [main] org.apache.druid.initialization.Initialization - Loading extension [druid-lookups-cached-single], jars: antlr-2.7.7.jar, mapdb-1.0.8.jar, druid-lookups-cached-single-0.21.0.jar, postgresql-42.2.14.jar, stringtemplate-3.2.jar, mysql-connector-java-5.1.48.jar ``` ```bash 2021-05-28T10:42:55,053 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/master/coordinator-overlord:lib/FastInfoset-1.2.15.jar:lib/log4j-core-2.8.2.jar:lib/druid-processing-0.21.0.jar:lib/httpclient-4.5.10.jar:lib/jetty-server-9.4.39.v20210325.jar:lib/druid-server-0.21.0.jar:lib/maven-model-3.1.1.jar:lib/plexus-interpolation-1.19.jar:lib/jackson-jaxrs-base-2.10.2.jar:lib/netty-3.10.6.Final.jar:lib/netty-codec-http-4.1.48.Final.jar:lib/httpcore-4.4.11.jar:lib/airline-0.7.jar:lib/calcite-linq4j-1.21.0.jar:lib/istack-commons-runtime-3.0.7.jar:lib/joda-time-2.10.5.jar:lib/resilience4j-core-1.3.1.jar:lib/jetty-servlets-9.4.39.v20210325.jar:lib/zookeeper-3.4.14.jar:lib/slf4j-api-1.7.25.jar:lib/netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:lib/jersey-servlet-1.19.3.jar:lib/jetty-util-ajax-9.4.39.v20210325.jar:lib/RoaringBitmap-0.9.0.jar:lib/commons-net-3.6.jar:lib/async-http-client-netty- utils-2.5.3.jar:lib/spymemcached-2.12.3.jar:lib/netty-handler-4.1.48.Final.jar:lib/curator-framework-4.3.0.jar:lib/jersey-guice-1.19.3.jar:lib/jaxb-runtime-2.3.1.jar:lib/caffeine-2.8.0.jar:lib/jcl-over-slf4j-1.7.12.jar:lib/icu4j-55.1.jar:lib/commons-dbcp2-2.0.1.jar:lib/druid-indexing-service-0.21.0.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/audience-annotations-0.5.0.jar:lib/javax.servlet-api-3.1.0.jar:lib/config-magic-0.9.jar:lib/druid-sql-0.21.0.jar:lib/jsr305-2.0.1.jar:lib/jetty-client-9.4.39.v20210325.jar:lib/jetty-util-9.4.39.v20210325.jar:lib/commons-math3-3.6.1.jar:lib/maven-settings-3.1.1.jar:lib/jvm-attach-api-1.5.jar:lib/avatica-core-1.17.0.jar:lib/netty-codec-4.1.48.Final.jar:lib/derbynet-10.14.2.0.jar:lib/jersey-core-1.19.3.jar:lib/vavr-0.10.2.jar:lib/j2objc-annotations-1.1.jar:lib/jcodings-1.0.43.jar:lib/google-api-client-1.26.0.jar:lib/commons-compress-1.19.jar:lib/maven-settings-builder-3.1.1.jar:lib/antlr4-runtime-4.5.1.jar:lib/netty-common-4.1.48.Final.jar:lib/g oogle-http-client-jackson2-1.26.0.jar:lib/log4j-jul-2.8.2.jar:lib/aether-impl-0.9.0.M2.jar:lib/checker-qual-2.5.7.jar:lib/druid-indexing-hadoop-0.21.0.jar:lib/derby-10.14.2.0.jar:lib/commons-io-2.6.jar:lib/commons-logging-1.1.1.jar:lib/curator-recipes-4.3.0.jar:lib/jna-4.5.1.jar:lib/jersey-server-1.19.3.jar:lib/maven-aether-provider-3.1.1.jar:lib/datasketches-memory-1.2.0-incubating.jar:lib/netty-handler-proxy-4.1.48.Final.jar:lib/vavr-match-0.10.2.jar:lib/google-oauth-client-1.26.0.jar:lib/error_prone_annotations-2.3.2.jar:lib/netty-codec-dns-4.1.48.Final.jar:lib/json-path-2.3.0.jar:lib/guice-multibindings-4.1.0.jar:lib/commons-text-1.3.jar:lib/rhino-1.7.11.jar:lib/log4j-1.2-api-2.8.2.jar:lib/druid-aws-common-0.21.0.jar:lib/jackson-jq-0.0.10.jar:lib/jackson-module-jaxb-annotations-2.10.2.jar:lib/commons-lang3-3.8.1.jar:lib/commons-compiler-3.0.11.jar:lib/datasketches-java-1.3.0-incubating.jar:lib/jetty-http-9.4.39.v20210325.jar:lib/netty-buffer-4.1.48.Final.jar:lib/maven-model-buil der-3.1.1.jar:lib/aws-java-sdk-kms-1.11.199.jar:lib/txw2-2.3.1.jar:lib/maven-repository-metadata-3.1.1.jar:lib/calcite-core-1.21.0.jar:lib/netty-transport-native-unix-common-4.1.48.Final.jar:lib/zstd-jni-1.3.3-1.jar:lib/curator-x-discovery-4.3.0.jar:lib/jakarta.activation-api-1.2.1.jar:lib/commons-lang-2.6.jar:lib/asm-7.1.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/avatica-metrics-1.17.0.jar:lib/druid-hll-0.21.0.jar:lib/commons-collections4-4.2.jar:lib/derbyclient-10.14.2.0.jar:lib/google-http-client-1.26.0.jar:lib/log4j-api-2.8.2.jar:lib/hibernate-validator-5.2.5.Final.jar:lib/druid-gcp-common-0.21.0.jar:lib/joni-2.1.27.jar:lib/jackson-dataformat-smile-2.10.2.jar:lib/aws-java-sdk-core-1.11.199.jar:lib/guava-16.0.1.jar:lib/plexus-utils-3.0.24.jar:lib/netty-resolver-4.1.48.Final.jar:lib/javax.el-3.0.0.jar:lib/jmespath-java-1.11.199.jar:lib/jaxb-api-2.3.1.jar:lib/xz-1.8.jar:lib/tesla-aether-0.0.5.jar:lib/validation-api-1.1.0.Final.jar:lib/avatica-server-1.17.0.jar:lib/jackson-annota tions-2.10.2.jar:lib/jackson-module-guice-2.10.2.jar:lib/aether-spi-0.9.0.M2.jar:lib/shims-0.9.0.jar:lib/disruptor-3.3.6.jar:lib/fastutil-8.2.3.jar:lib/cron-scheduler-0.1.jar:lib/commons-collections-3.2.2.jar:lib/accessors-smart-1.2.jar:lib/jdbi-2.63.1.jar:lib/druid-console-0.21.0.jar:lib/jackson-core-2.10.2.jar:lib/commons-beanutils-1.9.4.jar:lib/javax.activation-1.2.0.jar:lib/druid-core-0.21.0.jar:lib/sigar-1.6.5.132.jar:lib/aether-api-0.9.0.M2.jar:lib/netty-reactive-streams-2.0.0.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/netty-transport-4.1.48.Final.jar:lib/javax.el-api-3.0.0.jar:lib/async-http-client-2.5.3.jar:lib/curator-client-4.3.0.jar:lib/compress-lzf-1.0.4.jar:lib/extendedset-0.21.0.jar:lib/javax.activation-api-1.2.0.jar:lib/guice-servlet-4.1.0.jar:lib/jetty-io-9.4.39.v20210325.jar:lib/ion-java-1.0.2.jar:lib/guice-assistedinject-4.1.0.jar:lib/asm-analysis-7.1.jar:lib/jackson-dataformat-cbor-2.10.2.jar:lib/netty-resolver-dns-4.1.48.Final.jar:lib/jetty-rewrite-9.4.39.v202103 25.jar:lib/commons-codec-1.13.jar:lib/janino-3.0.11.jar:lib/classmate-1.1.0.jar:lib/commons-pool2-2.2.jar:lib/netty-codec-socks-4.1.48.Final.jar:lib/esri-geometry-api-2.2.0.jar:lib/lz4-java-1.7.1.jar:lib/protobuf-java-3.11.0.jar:lib/log4j-slf4j-impl-2.8.2.jar:lib/jetty-servlet-9.4.39.v20210325.jar:lib/json-smart-2.3.jar:lib/aws-java-sdk-ec2-1.11.199.jar:lib/jboss-logging-3.2.1.Final.jar:lib/jackson-jaxrs-smile-provider-2.10.2.jar:lib/reactive-streams-1.0.2.jar:lib/javax.inject-1.jar:lib/aether-util-0.9.0.M2.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/jetty-proxy-9.4.39.v20210325.jar:lib/jline-0.9.94.jar:lib/jackson-datatype-guava-2.10.2.jar:lib/jackson-jaxrs-json-provider-2.10.2.jar:lib/aws-java-sdk-s3-1.11.199.jar:lib/wagon-provider-api-2.4.jar:lib/opencsv-4.6.jar:lib/jetty-security-9.4.39.v20210325.jar:lib/jsr311-api-1.1.1.jar:lib/jetty-continuation-9.4.39.v20210325.jar:lib/maven-artifact-3.6.0.jar:lib/druid-services-0.21.0.jar:lib/metrics-core-4.0.0.jar:lib/asm-tree-7.1.jar:lib/j ackson-datatype-joda-2.10.2.jar:lib/jackson-databind-2.10.5.1.jar:lib/asm-commons-7.1.jar:lib/stax-ex-1.8.jar:lib/aggdesigner-algorithm-6.0.jar:lib/guice-4.1.0.jar:lib/aopalliance-1.0.jar:lib/okhttp-1.0.2.jar:lib/mysql-connector-java-5.1.48.jar: ``` However, we still get the error: ```bash ��error�Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: Failed to find MySQL driver class. Please check the MySQL connector version 5.1.48 is in the classpath at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: -1, column: 354] (through reference chain: org.apache.druid.query.lookup.LookupsState["toLoad"]->java.util.LinkedHashMap["TEST_TOURNAMENT_LOOKUP_LOCAL_MYSQL"]->org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]->org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])��]} org.apache.druid.java.util.common.IOE: Bad update request to [http://10.1.11.240:8083/druid/listen/v1/lookups/updates] : [400] : [Bad Request] Response: [:) ��error�Cannot construct instance of `org.apache.druid.query.lookup.namespace.JdbcExtractionNamespace`, problem: Failed to find MySQL driver class. Please check the MySQL connector version 5.1.48 is in the classpath at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: -1, column: 354] (through reference chain: org.apache.druid.query.lookup.LookupsState["toLoad"]->java.util.LinkedHashMap["TEST_TOURNAMENT_LOOKUP_LOCAL_MYSQL"]->org.apache.druid.query.lookup.LookupExtractorFactoryContainer["lookupExtractorFactory"]->org.apache.druid.query.lookup.NamespaceLookupExtractorFactory["extractionNamespace"])��] at org.apache.druid.server.lookup.cache.LookupCoordinatorManager$LookupsCommunicator.updateNode(LookupCoordinatorManager.java:834) ~[druid-server-0.21.0.jar:0.21.0] at org.apache.druid.server.lookup.cache.LookupCoordinatorManager.doLookupManagementOnNode(LookupCoordinatorManager.java:663) ~[druid-server-0.21.0.jar:0.21.0] at org.apache.druid.server.lookup.cache.LookupCoordinatorManager.lambda$lookupManagementLoop$2(LookupCoordinatorManager.java:590) ~[druid-server-0.21.0.jar:0.21.0] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_275] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_275] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_275] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275] ``` All these logs above come from the coordinator pod, I can see the `MYSQL` jar available is indeed available under: ```bash lib/ extensions/druid-lookups-cached-global/ ls -lah extensions/druid-lookups-cached-single/ ``` Which confirms the params: ```yaml # Extension druid.extensions.loadList = '["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-s3-extensions", "druid-kafka-indexing-service", "druid-basic-security", "druid-lookups-cached-global", "druid-lookups-cached-single"]' # Lookups druid.access.jdbc.enforceAllowedProperties = true druid.access.jdbc_allowedProperties = '["useSSL", "requireSSL", "ssl", "sslmode"]' druid.access.jdbc_allowUnknownJdbcUrlFormat = true druid.lookup.lookupTierIsDatasource = false druid.lookup.lookupTier: __default ``` Based on the link: https://druid.apache.org/docs/latest/configuration/index.html I have also been able to create a fully `local` `k8s` deployment, with a `MYSQL` `docker` running outside of the cluster, to replicate the cloud environment, where I can debug this in realtime. I went more steps further and investigated the source code: * https://github.com/apache/druid/blob/cfcebc40f67a38eb0a3a23ae8919100bd432d533/extensions-core/lookups-cached-global/src/main/java/org/apache/druid/query/lookup/namespace/JdbcExtractionNamespace.java#L127 ```java NonRegisteringDriver driver = new NonRegisteringDriver(); ``` Where is seems this fails, and throws the error, I made sure that `NonRegisteringDriver` do exist in the `JAR`: ```bash jar tf mysql-connector-java-5.1.48.jar ``` ```bash com/mysql/jdbc/NonRegisteringDriver.class ``` I would be very happy, if somebody can explain to me why this does not work. What is missing here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
