Hi Vova, Thanks for the response. I took a look at the ES-Hadoop github repo and actually found two old issues dealing with this very problem:
https://github.com/elastic/elasticsearch-hadoop/issues/932 <https://github.com/elastic/elasticsearch-hadoop/issues/932> https://github.com/elastic/elasticsearch-hadoop/issues/1184 <https://github.com/elastic/elasticsearch-hadoop/issues/1184> It looks like all these issues were complex and nobody submitted one with a simple "update the library" PR, so I'm going to try to do that and see what happens. From what I understand, ES doesn't always like or approve of PRs from outside contributors even though it is an open source project. Best, -- C > On Apr 25, 2020, at 7:08 AM, Vova Vysotskyi <[email protected]> wrote: > > Hi all, > > Thanks, Charles, for sharing additional info. I have checked out to your > branch, and now I see that elasticsearch-hadoop uses classes from the > commons-httpclient library. > Since ES is an open-source project, could you please create an issue for > them to update this library? > For now, I agree with Paul regarding retaining the older version. > > When I've excluded commons-httpclient, I had a similar issue > with nl.basjes.parse.httpdlog:httpdlog-parser, but after I have reported an > issue ticket regarding updating this library, Niels rapidly fixed it and > released a new version (thanks a lot to Niels for this!). > > Kind regards, > Volodymyr Vysotskyi > > > On Fri, Apr 24, 2020 at 5:58 AM Paul Rogers <[email protected]> > wrote: > >> Hi All, >> I think there may be a bit of confusion. It may be true that some of >> Drill's dependencies now use the newer version of the library >> httpcomponents:httpclient. However, it looks like ES directly depends on >> the older flavor. >> >> We have pom file entries which exclude that old version. As a result, ES >> can't find the older version at run time. >> >> So, maybe two choices. 1) convince ES to use the newer version (how?), or >> 2 retain the older version (don't exclude it.) >> >> Would sure be nice if each storage plugin could run in its own class >> loader to avoid these issues. We're slowly moving in that direction. >> >> Thanks, >> - Paul >> >> >> >> On Thursday, April 23, 2020, 5:35:55 PM PDT, Charles Givre < >> [email protected]> wrote: >> >> Hi Vova, >> Thanks for the response. I've been slowly poking at a storage plugin for >> ElasticSearch.[1] I was going to do some work on it, but after rebasing to >> the latest master, I'm getting errors in my unit tests that were not there >> before. >> >> Here's the relevant snippet of the dependency tree: >> >> [INFO] +- >> org.elasticsearch.client:elasticsearch-rest-client:jar:7.6.2:compile >> [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile >> [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile >> [INFO] | \- org.apache.httpcomponents:httpcore-nio:jar:4.4.12:compile >> [INFO] +- org.elasticsearch:elasticsearch-hadoop:jar:7.6.2:compile >> >> Currently, I don't have the dependency excluded or anything like that in >> the pom.xml for the storage plugin so I would assume that the dependency >> would be included, but it doesn't seem to be. Would you have any >> suggestions as to how to fix it? >> >> >> Thanks, >> -- C >> >> >> >> Here's the full stack trace: >> 19:58:51.940 [Time-limited test] DEBUG o.a.d.e.s.e.TestElasticQueries - >> select * from elasticsearch.employee.`developer` >> 19:58:52.011 [215dd444-5b82-78b6-6222-25db75b0a934:foreman] DEBUG >> o.a.d.e.s.e.ElasticSearchGroupScan - Getting region locations >> >> org.apache.drill.exec.rpc.RpcException: >> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: >> ClassNotFoundException: >> org.apache.commons.httpclient.protocol.ProtocolSocketFactory >> >> >> Please, refer to logs for more information. >> >> [Error Id: f43ab4ec-d557-45df-a41d-1a0221c3ffdb on 192.168.1.25:31013] >> >> at >> org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:59) >> at >> org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:881) >> at >> org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:583) >> at org.apache.drill.test.QueryBuilder.results(QueryBuilder.java:331) >> at >> org.apache.drill.test.ClusterFixture$FixtureTestServices.testRunAndReturn(ClusterFixture.java:615) >> at >> org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:938) >> at >> org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:533) >> at >> org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:172) >> at org.apache.drill.test.TestBuilder.go(TestBuilder.java:145) >> at >> org.apache.drill.exec.store.elasticsearch.TestElasticQueries.testSimpleStarQuery(TestElasticQueries.java:83) >> at java.lang.Thread.run(Thread.java:748) >> Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM >> ERROR: ClassNotFoundException: >> org.apache.commons.httpclient.protocol.ProtocolSocketFactory >> >> >> Please, refer to logs for more information. >> >> [Error Id: f43ab4ec-d557-45df-a41d-1a0221c3ffdb on 192.168.1.25:31013] >> at >> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:125) >> at >> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422) >> at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96) >> at >> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) >> at >> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) >> at >> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >> at >> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >> at >> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >> at >> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) >> at >> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) >> at >> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) >> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) >> at >> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) >> ... 1 more >> Caused by: org.apache.drill.exec.work.foreman.ForemanException: >> Unexpected exception during fragment initialization: >> org/apache/commons/httpclient/protocol/ProtocolSocketFactory >> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:301) >> at .......(:0) >> Caused by: java.lang.Error: >> org/apache/commons/httpclient/protocol/ProtocolSocketFactory >> at >> org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory.create(CommonsHttpTransportFactory.java:40) >> at >> org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:99) >> at >> org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:82) >> at >> org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:58) >> at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:101) >> at >> org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:327) >> at >> org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:220) >> at >> org.apache.drill.exec.store.elasticsearch.ElasticSearchGroupScan.init(ElasticSearchGroupScan.java:196) >> at >> org.apache.drill.exec.store.elasticsearch.ElasticSearchGroupScan.<init>(ElasticSearchGroupScan.java:124) >> at >> org.apache.drill.exec.store.elasticsearch.ElasticSearchStoragePlugin.getPhysicalScan(ElasticSearchStoragePlugin.java:122) >> at >> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:112) >> at >> org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:117) >> at >> org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:119) >> at >> org.apache.drill.exec.planner.common.DrillScanRelBase.<init>(DrillScanRelBase.java:51) >> at >> org.apache.drill.exec.planner.logical.DrillScanRel.<init>(DrillScanRel.java:79) >> at >> org.apache.drill.exec.planner.logical.DrillScanRel.<init>(DrillScanRel.java:66) >> at >> org.apache.drill.exec.planner.logical.DrillScanRel.<init>(DrillScanRel.java:59) >> at >> org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:38) >> at >> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) >> at >> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:633) >> at >> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:405) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:351) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:245) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:308) >> at >> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:173) >> at >> org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283) >> at >> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163) >> at >> org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128) >> at >> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93) >> at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593) >> at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274) >> ... 1 more >> Caused by: java.lang.ClassNotFoundException: >> org.apache.commons.httpclient.protocol.ProtocolSocketFactory >> at .......(:0) >> ... 33 more >> >> >> Here is the dependency tree: >> >> (base) ➜ storage-elastic git:(storage-elastic2) ✗ mvn dependency:tree >> [INFO] Scanning for projects... >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Detecting the operating system and CPU architecture >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] os.detected.name: osx >> [INFO] os.detected.arch: x86_64 >> [INFO] os.detected.version: 10.15 >> [INFO] os.detected.version.major: 10 >> [INFO] os.detected.version.minor: 15 >> [INFO] os.detected.classifier: osx-x86_64 >> [INFO] >> [INFO] --------< org.apache.drill.contrib:drill-storage-elasticsearch >>> -------- >> [INFO] Building contrib/storage-elasticsearch 1.18.0-SNAPSHOT >> [INFO] --------------------------------[ jar >> ]--------------------------------- >> [INFO] >> [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ >> drill-storage-elasticsearch --- >> [INFO] >> org.apache.drill.contrib:drill-storage-elasticsearch:jar:1.18.0-SNAPSHOT >> [INFO] +- org.apache.drill.exec:drill-java-exec:jar:1.18.0-SNAPSHOT:compile >> [INFO] | +- org.apache.httpcomponents:httpasyncclient:jar:4.1.4:compile >> [INFO] | +- org.ow2.asm:asm-commons:jar:7.3.1:compile >> [INFO] | | +- org.ow2.asm:asm:jar:7.3.1:compile >> [INFO] | | +- org.ow2.asm:asm-tree:jar:7.3.1:compile >> [INFO] | | \- org.ow2.asm:asm-analysis:jar:7.3.1:compile >> [INFO] | +- org.ow2.asm:asm-util:jar:7.3.1:compile >> [INFO] | +- org.apache.commons:commons-pool2:jar:2.6.0:compile >> [INFO] | +- com.univocity:univocity-parsers:jar:2.8.3:compile >> [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile >> [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.5.6:compile >> [INFO] | +- xerces:xercesImpl:jar:2.12.0:compile >> [INFO] | | \- xml-apis:xml-apis:jar:1.4.01:compile >> [INFO] | +- xalan:xalan:jar:2.7.2:compile >> [INFO] | | \- xalan:serializer:jar:2.7.2:compile >> [INFO] | +- com.sun.codemodel:codemodel:jar:2.6:compile >> [INFO] | +- org.eclipse.jetty:jetty-server:jar:9.3.28.v20191105:compile >> [INFO] | | +- javax.servlet:javax.servlet-api:jar:3.1.0:compile >> [INFO] | | +- org.eclipse.jetty:jetty-http:jar:9.3.28.v20191105:compile >> [INFO] | | \- org.eclipse.jetty:jetty-io:jar:9.3.28.v20191105:compile >> [INFO] | +- org.eclipse.jetty:jetty-servlet:jar:9.3.28.v20191105:compile >> [INFO] | | \- >> org.eclipse.jetty:jetty-security:jar:9.3.28.v20191105:compile >> [INFO] | +- org.eclipse.jetty:jetty-servlets:jar:9.3.28.v20191105:compile >> [INFO] | +- >> org.glassfish.jersey.containers:jersey-container-jetty-servlet:jar:2.25.1:compile >> [INFO] | | +- >> org.glassfish.jersey.containers:jersey-container-servlet:jar:2.25.1:compile >> [INFO] | | | \- >> org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:compile >> [INFO] | | +- >> org.glassfish.jersey.containers:jersey-container-jetty-http:jar:2.25.1:compile >> [INFO] | | | +- >> org.glassfish.hk2.external:javax.inject:jar:2.5.0-b32:compile >> [INFO] | | | \- >> org.eclipse.jetty:jetty-continuation:jar:9.2.14.v20151106:compile >> [INFO] | | +- org.eclipse.jetty:jetty-webapp:jar:9.3.28.v20191105:compile >> [INFO] | | | \- org.eclipse.jetty:jetty-xml:jar:9.3.28.v20191105:compile >> [INFO] | | +- org.glassfish.jersey.core:jersey-common:jar:2.25.1:compile >> [INFO] | | | +- >> org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.25.1:compile >> [INFO] | | | +- org.glassfish.hk2:hk2-api:jar:2.5.0-b32:compile >> [INFO] | | | | +- org.glassfish.hk2:hk2-utils:jar:2.5.0-b32:compile >> [INFO] | | | | \- >> org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b32:compile >> [INFO] | | | +- org.glassfish.hk2:hk2-locator:jar:2.5.0-b32:compile >> [INFO] | | | \- >> org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile >> [INFO] | | +- org.glassfish.jersey.core:jersey-server:jar:2.25.1:compile >> [INFO] | | | +- >> org.glassfish.jersey.core:jersey-client:jar:2.25.1:compile >> [INFO] | | | +- >> org.glassfish.jersey.media:jersey-media-jaxb:jar:2.25.1:compile >> [INFO] | | | \- javax.validation:validation-api:jar:1.1.0.Final:compile >> [INFO] | | \- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile >> [INFO] | +- >> org.glassfish.jersey.media:jersey-media-multipart:jar:2.25.1:compile >> [INFO] | | \- org.jvnet.mimepull:mimepull:jar:1.9.6:compile >> [INFO] | +- >> org.glassfish.jersey.media:jersey-media-json-jackson:jar:2.25.1:compile >> [INFO] | | +- >> org.glassfish.jersey.ext:jersey-entity-filtering:jar:2.25.1:compile >> [INFO] | | \- >> com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.8.4:compile >> [INFO] | +- >> com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.10.3:compile >> [INFO] | | \- >> com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.3:compile >> [INFO] | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile >> [INFO] | | \- >> jakarta.activation:jakarta.activation-api:jar:1.2.1:compile >> [INFO] | +- org.mongodb:mongo-java-driver:jar:3.8.0:compile >> [INFO] | +- >> com.fasterxml.jackson.module:jackson-module-afterburner:jar:2.10.3:compile >> [INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.10.3:compile >> [INFO] | +- >> org.honton.chas.hocon:jackson-dataformat-hocon:jar:1.1.1:compile >> [INFO] | +- >> org.glassfish.jersey.ext:jersey-mvc-freemarker:jar:2.25.1:compile >> [INFO] | | \- org.glassfish.jersey.ext:jersey-mvc:jar:2.25.1:compile >> [INFO] | +- >> com.github.vvysotskyi.drill-calcite:calcite-core:jar:1.21.0-drill-r0:compile >> [INFO] | | +- >> com.github.vvysotskyi.drill-calcite:calcite-linq4j:jar:1.21.0-drill-r0:compile >> [INFO] | | +- org.apache.commons:commons-dbcp2:jar:2.6.0:compile >> [INFO] | | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:compile >> [INFO] | | +- >> com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.9.9:compile >> [INFO] | | | \- org.yaml:snakeyaml:jar:1.26:compile >> [INFO] | | +- com.google.guava:guava:jar:19.0:compile >> [INFO] | | +- com.yahoo.datasketches:sketches-core:jar:0.9.0:compile >> [INFO] | | | \- com.yahoo.datasketches:memory:jar:0.9.0:compile >> [INFO] | | +- com.jayway.jsonpath:json-path:jar:2.4.0:compile >> [INFO] | | \- net.hydromatic:aggdesigner-algorithm:jar:6.0:compile >> [INFO] | | \- commons-lang:commons-lang:jar:2.4:compile >> [INFO] | +- org.apache.calcite.avatica:avatica:jar:1.15.0:compile >> [INFO] | +- net.sf.jpam:jpam:jar:1.1:compile >> [INFO] | +- org.bouncycastle:bcpkix-jdk15on:jar:1.60:compile >> [INFO] | | \- org.bouncycastle:bcprov-jdk15on:jar:1.60:compile >> [INFO] | +- org.freemarker:freemarker:jar:2.3.28:compile >> [INFO] | +- org.apache.parquet:parquet-column:jar:1.11.0:compile >> [INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.11.0:compile >> [INFO] | +- org.apache.parquet:parquet-hadoop:jar:1.11.0:compile >> [INFO] | | +- >> org.apache.parquet:parquet-format-structures:jar:1.11.0:compile >> [INFO] | | +- org.apache.parquet:parquet-jackson:jar:1.11.0:compile >> [INFO] | | \- commons-pool:commons-pool:jar:1.6:compile >> [INFO] | +- org.apache.parquet:parquet-format:jar:2.8.0:compile >> [INFO] | | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile >> [INFO] | +- org.apache.parquet:parquet-common:jar:1.11.0:compile >> [INFO] | | \- org.apache.yetus:audience-annotations:jar:0.11.0:compile >> [INFO] | +- javax.inject:javax.inject:jar:1:compile >> [INFO] | +- org.apache.drill:drill-protocol:jar:1.18.0-SNAPSHOT:compile >> [INFO] | | +- io.protostuff:protostuff-core:jar:1.7.1:compile >> [INFO] | | | \- io.protostuff:protostuff-api:jar:1.7.1:compile >> [INFO] | | \- io.protostuff:protostuff-json:jar:1.7.1:compile >> [INFO] | +- org.apache.drill:drill-common:jar:1.18.0-SNAPSHOT:compile >> [INFO] | +- org.apache.drill:drill-logical:jar:1.18.0-SNAPSHOT:compile >> [INFO] | | \- org.antlr:antlr4-runtime:jar:4.8-1:compile >> [INFO] | +- org.apache.drill.exec:drill-rpc:jar:1.18.0-SNAPSHOT:compile >> [INFO] | | \- >> io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.0.48.Final:compile >> [INFO] | +- >> org.apache.drill.memory:drill-memory-base:jar:1.18.0-SNAPSHOT:compile >> [INFO] | +- org.apache.drill.exec:vector:jar:1.18.0-SNAPSHOT:compile >> [INFO] | +- >> org.apache.drill.metastore:drill-metastore-api:jar:1.18.0-SNAPSHOT:compile >> [INFO] | | \- >> com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.10.3:compile >> [INFO] | +- com.beust:jcommander:jar:1.30:compile >> [INFO] | +- org.apache.curator:curator-client:jar:4.3.0:compile >> [INFO] | | \- org.apache.zookeeper:zookeeper:jar:3.5.7:compile >> [INFO] | | +- org.apache.zookeeper:zookeeper-jute:jar:3.5.7:compile >> [INFO] | | \- >> io.netty:netty-transport-native-epoll:jar:4.1.45.Final:compile >> [INFO] | | \- >> io.netty:netty-transport-native-unix-common:jar:4.1.45.Final:compile >> [INFO] | +- org.apache.curator:curator-framework:jar:4.3.0:compile >> [INFO] | +- org.apache.curator:curator-recipes:jar:4.3.0:compile >> [INFO] | +- org.apache.curator:curator-x-discovery:jar:4.3.0:compile >> [INFO] | +- com.carrotsearch:hppc:jar:0.7.1:compile >> [INFO] | +- com.google.protobuf:protobuf-java:jar:3.11.1:compile >> [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile >> [INFO] | +- joda-time:joda-time:jar:2.10.5:compile >> [INFO] | +- org.codehaus.janino:janino:jar:3.0.11:compile >> [INFO] | +- org.codehaus.janino:commons-compiler:jar:3.0.11:compile >> [INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:compile >> [INFO] | | \- it.unimi.dsi:fastutil:jar:6.5.7:compile >> [INFO] | +- com.tdunning:t-digest:jar:3.2:compile >> [INFO] | +- org.apache.hadoop:hadoop-common:jar:3.2.1:compile >> [INFO] | | +- org.apache.hadoop:hadoop-annotations:jar:3.2.1:compile >> [INFO] | | +- commons-cli:commons-cli:jar:1.4:compile >> [INFO] | | +- org.apache.commons:commons-math3:jar:3.1.1:compile >> [INFO] | | +- commons-net:commons-net:jar:3.6:compile >> [INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile >> [INFO] | | +- org.eclipse.jetty:jetty-util:jar:9.3.28.v20191105:compile >> [INFO] | | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime >> [INFO] | | +- com.sun.jersey:jersey-servlet:jar:1.19:compile >> [INFO] | | +- commons-beanutils:commons-beanutils:jar:1.9.4:compile >> [INFO] | | +- org.apache.commons:commons-configuration2:jar:2.1.1:compile >> [INFO] | | +- org.apache.commons:commons-text:jar:1.6:compile >> [INFO] | | +- com.google.re2j:re2j:jar:1.1:compile >> [INFO] | | +- com.google.code.gson:gson:jar:2.2.4:compile >> [INFO] | | +- org.apache.hadoop:hadoop-auth:jar:3.2.1:compile >> [INFO] | | +- com.jcraft:jsch:jar:0.1.54:compile >> [INFO] | | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:compile >> [INFO] | | +- org.apache.commons:commons-compress:jar:1.20:compile >> [INFO] | | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile >> [INFO] | | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile >> [INFO] | | \- dnsjava:dnsjava:jar:2.1.7:compile >> [INFO] | +- org.apache.hadoop:hadoop-client:jar:3.2.1:compile >> [INFO] | | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.2.1:compile >> [INFO] | | | \- com.squareup.okhttp:okhttp:jar:2.7.5:compile >> [INFO] | | | \- com.squareup.okio:okio:jar:1.6.0:compile >> [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:3.2.1:compile >> [INFO] | | +- >> org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.2.1:compile >> [INFO] | | | \- org.apache.hadoop:hadoop-yarn-common:jar:3.2.1:compile >> [INFO] | | | \- javax.xml.bind:jaxb-api:jar:2.2.11:compile >> [INFO] | | \- >> org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.2.1:compile >> [INFO] | | \- >> org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.2.1:compile >> [INFO] | +- org.apache.avro:avro:jar:1.9.1:compile >> [INFO] | +- org.apache.avro:avro-mapred:jar:1.9.1:compile >> [INFO] | | +- org.apache.avro:avro-ipc:jar:1.9.1:compile >> [INFO] | | | \- org.apache.velocity:velocity-engine-core:jar:2.0:compile >> [INFO] | | \- org.apache.avro:avro-ipc-jetty:jar:1.9.1:compile >> [INFO] | +- nl.basjes.parse.httpdlog:httpdlog-parser:jar:5.3:compile >> [INFO] | | +- nl.basjes.parse:parser-core:jar:5.3:compile >> [INFO] | | \- com.maxmind.geoip2:geoip2:jar:2.13.1:compile >> [INFO] | | \- com.maxmind.db:maxmind-db:jar:1.3.1:compile >> [INFO] | +- org.kohsuke:libpam4j:jar:1.8-rev2:compile >> [INFO] | | \- net.java.dev.jna:jna:jar:4.1.0:compile >> [INFO] | +- io.netty:netty-tcnative:jar:2.0.1.Final:runtime (optional) >> [INFO] | +- com.drewnoakes:metadata-extractor:jar:2.13.0:compile >> [INFO] | | \- com.adobe.xmp:xmpcore:jar:6.0.6:compile >> [INFO] | +- fr.bmartel:pcapngdecoder:jar:1.2:compile >> [INFO] | \- sqlline:sqlline:jar:1.9.0:compile >> [INFO] | +- org.jline:jline-terminal:jar:3.12.1:compile >> [INFO] | +- org.jline:jline-reader:jar:3.12.1:compile >> [INFO] | +- org.jline:jline-terminal-jansi:jar:3.12.1:compile >> [INFO] | | \- org.fusesource.jansi:jansi:jar:1.18:compile >> [INFO] | +- org.jline:jline-terminal-jna:jar:3.12.1:compile >> [INFO] | \- org.jline:jline-builtins:jar:3.12.1:compile >> [INFO] | \- org.jline:jline-style:jar:3.12.1:compile >> [INFO] +- >> org.elasticsearch.client:elasticsearch-rest-client:jar:7.6.2:compile >> [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile >> [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile >> [INFO] | \- org.apache.httpcomponents:httpcore-nio:jar:4.4.12:compile >> [INFO] +- org.elasticsearch:elasticsearch-hadoop:jar:7.6.2:compile >> [INFO] +- org.elasticsearch.client:transport:jar:7.6.2:compile >> [INFO] | +- org.elasticsearch:elasticsearch:jar:7.6.2:compile >> [INFO] | | +- org.elasticsearch:elasticsearch-core:jar:7.6.2:compile >> [INFO] | | +- org.elasticsearch:elasticsearch-secure-sm:jar:7.6.2:compile >> [INFO] | | +- org.elasticsearch:elasticsearch-x-content:jar:7.6.2:compile >> [INFO] | | | +- >> com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.8.11:compile >> [INFO] | | | \- >> com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.11:compile >> [INFO] | | +- org.elasticsearch:elasticsearch-geo:jar:7.6.2:compile >> [INFO] | | +- org.apache.lucene:lucene-core:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-analyzers-common:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-backward-codecs:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-grouping:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-highlighter:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-join:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-memory:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-misc:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-queries:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-queryparser:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-sandbox:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-spatial:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-spatial-extras:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-spatial3d:jar:8.4.0:compile >> [INFO] | | +- org.apache.lucene:lucene-suggest:jar:8.4.0:compile >> [INFO] | | +- org.elasticsearch:elasticsearch-cli:jar:7.6.2:compile >> [INFO] | | | \- net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile >> [INFO] | | +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile >> [INFO] | | +- org.apache.logging.log4j:log4j-api:jar:2.11.1:compile >> [INFO] | | \- org.elasticsearch:jna:jar:4.5.1:compile >> [INFO] | +- >> org.elasticsearch.plugin:transport-netty4-client:jar:7.6.2:compile >> [INFO] | | +- io.netty:netty-codec-http:jar:4.1.43.Final:compile >> [INFO] | | \- io.netty:netty-resolver:jar:4.1.43.Final:compile >> [INFO] | +- org.elasticsearch.plugin:reindex-client:jar:7.6.2:compile >> [INFO] | | \- >> org.elasticsearch:elasticsearch-ssl-config:jar:7.6.2:compile >> [INFO] | +- >> org.elasticsearch.plugin:lang-mustache-client:jar:7.6.2:compile >> [INFO] | | \- >> com.github.spullara.mustache.java:compiler:jar:0.9.6:compile >> [INFO] | +- org.elasticsearch.plugin:percolator-client:jar:7.6.2:compile >> [INFO] | +- org.elasticsearch.plugin:parent-join-client:jar:7.6.2:compile >> [INFO] | \- org.elasticsearch.plugin:rank-eval-client:jar:7.6.2:compile >> [INFO] +- >> org.apache.drill.exec:drill-java-exec:jar:tests:1.18.0-SNAPSHOT:test >> [INFO] +- org.apache.drill:drill-common:jar:tests:1.18.0-SNAPSHOT:test >> [INFO] | +- com.typesafe:config:jar:1.0.0:compile >> [INFO] | +- org.apache.commons:commons-lang3:jar:3.10:compile >> [INFO] | +- org.msgpack:msgpack:jar:0.6.6:compile >> [INFO] | | \- com.googlecode.json-simple:json-simple:jar:1.1.1:compile >> [INFO] | +- org.reflections:reflections:jar:0.9.10:compile >> [INFO] | | \- com.google.code.findbugs:annotations:jar:2.0.1:compile >> [INFO] | +- org.javassist:javassist:jar:3.27.0-GA:compile >> [INFO] | +- >> com.fasterxml.jackson.core:jackson-annotations:jar:2.10.3:compile >> [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.10.3:compile >> [INFO] | +- io.dropwizard.metrics:metrics-core:jar:4.0.2:compile >> [INFO] | +- io.dropwizard.metrics:metrics-servlets:jar:4.0.2:compile >> [INFO] | | +- >> io.dropwizard.metrics:metrics-healthchecks:jar:4.0.2:compile >> [INFO] | | +- io.dropwizard.metrics:metrics-json:jar:4.0.2:compile >> [INFO] | | \- com.papertrail:profiler:jar:1.0.2:compile >> [INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:4.0.2:compile >> [INFO] | \- io.dropwizard.metrics:metrics-jmx:jar:4.0.2:compile >> [INFO] +- commons-codec:commons-codec:jar:1.14:compile >> [INFO] +- io.netty:netty-handler:jar:4.0.48.Final:compile >> [INFO] | +- io.netty:netty-buffer:jar:4.0.48.Final:compile >> [INFO] | +- io.netty:netty-transport:jar:4.0.48.Final:compile >> [INFO] | \- io.netty:netty-codec:jar:4.0.48.Final:compile >> [INFO] +- io.netty:netty-common:jar:4.0.48.Final:compile >> [INFO] +- org.apache.drill:drill-shaded-guava:jar:28.2-jre:compile >> [INFO] | +- >> com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile >> [INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile >> [INFO] | +- org.checkerframework:checker-qual:jar:2.10.0:compile >> [INFO] | +- >> com.google.errorprone:error_prone_annotations:jar:2.3.4:compile >> [INFO] | \- com.google.j2objc:j2objc-annotations:jar:1.3:compile >> [INFO] +- org.slf4j:slf4j-api:jar:1.7.26:compile >> [INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.26:compile >> [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.26:compile >> [INFO] +- org.slf4j:log4j-over-slf4j:jar:1.7.26:compile >> [INFO] +- commons-io:commons-io:jar:2.4:compile >> [INFO] +- org.jmockit:jmockit:jar:1.47:test >> [INFO] +- junit:junit:jar:4.12:test >> [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test >> [INFO] +- org.mockito:mockito-core:jar:2.23.4:test >> [INFO] | +- net.bytebuddy:byte-buddy:jar:1.9.3:test >> [INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.9.3:test >> [INFO] | \- org.objenesis:objenesis:jar:2.6:test >> [INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:test >> [INFO] +- ch.qos.logback:logback-core:jar:1.2.3:test >> [INFO] +- >> de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic:jar:0.9.44:test >> [INFO] | +- de.huxhorn.sulky:de.huxhorn.sulky.codec:jar:0.9.17:test >> [INFO] | | \- de.huxhorn.sulky:de.huxhorn.sulky.io:jar:0.9.17:test >> [INFO] | +- >> de.huxhorn.lilith:de.huxhorn.lilith.logback.converter-classic:jar:0.9.44:test >> [INFO] | | +- >> de.huxhorn.lilith:de.huxhorn.lilith.data.converter:jar:0.9.44:test >> [INFO] | | +- >> de.huxhorn.lilith:de.huxhorn.lilith.logback.classic:jar:0.9.44:test >> [INFO] | | \- >> de.huxhorn.lilith:de.huxhorn.lilith.data.logging:jar:0.9.44:test >> [INFO] | | +- >> de.huxhorn.sulky:de.huxhorn.sulky.formatting:jar:0.9.17:test >> [INFO] | | \- >> de.huxhorn.lilith:de.huxhorn.lilith.data.eventsource:jar:0.9.44:test >> [INFO] | +- >> de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-core:jar:0.9.44:test >> [INFO] | | \- de.huxhorn.lilith:de.huxhorn.lilith.sender:jar:0.9.44:test >> [INFO] | \- >> de.huxhorn.lilith:de.huxhorn.lilith.data.logging.protobuf:jar:0.9.44:test >> [INFO] +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile >> [INFO] +- org.apache.kerby:kerb-client:jar:1.0.0:test >> [INFO] | +- org.apache.kerby:kerby-config:jar:1.0.0:test >> [INFO] | +- org.apache.kerby:kerb-common:jar:1.0.0:test >> [INFO] | | \- org.apache.kerby:kerb-crypto:jar:1.0.0:test >> [INFO] | \- org.apache.kerby:kerb-util:jar:1.0.0:test >> [INFO] +- org.apache.kerby:kerb-core:jar:1.0.0:test >> [INFO] | \- org.apache.kerby:kerby-pkix:jar:1.0.0:test >> [INFO] | +- org.apache.kerby:kerby-asn1:jar:1.0.0:test >> [INFO] | \- org.apache.kerby:kerby-util:jar:1.0.0:test >> [INFO] \- org.apache.kerby:kerb-simplekdc:jar:1.0.0:test >> [INFO] \- org.apache.kerby:kerb-admin:jar:1.0.0:test >> [INFO] +- org.apache.kerby:kerb-server:jar:1.0.0:test >> [INFO] | \- org.apache.kerby:kerb-identity:jar:1.0.0:test >> [INFO] \- org.apache.kerby:kerby-xdr:jar:1.0.0:test >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] BUILD SUCCESS >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 1.402 s >> [INFO] Finished at: 2020-04-23T20:30:50-04:00 >> [INFO] >> ------------------------------------------------------------------------ >> (base) ➜ storage-elastic git:(storage-elastic2) ✗ >> >> >> >> >> >> >> [1]: https://github.com/apache/drill/pull/1961 < >> https://github.com/apache/drill/pull/1961> >> >> >> >> >>> On Apr 22, 2020, at 11:11 AM, Vova Vysotskyi <[email protected]> wrote: >>> >>> Hi Charles, >>> >>> Could you please share more details about your problem, for example, >> which >>> one dependency uses commons-httpclient, which errors you see, etc. >>> >>> Kind regards, >>> Volodymyr Vysotskyi >>> >>> >>> On Wed, Apr 22, 2020 at 5:23 PM Charles Givre <[email protected]> wrote: >>> >>>> Hi Volodymyr, >>>> Thanks for the response. Do you have any suggestions as to how to fix >>>> this? I don't really know enough Maven to fix this. I've tried >> manually >>>> including the library in the pom, removing exclusions etc. >>>> Thanks, >>>> -- C >>>> >>>> >>>> >>>>> On Apr 20, 2020, at 4:15 AM, Vova Vysotskyi <[email protected]> wrote: >>>>> >>>>> Hi Charles, >>>>> >>>>> *commons-httpclient* was excluded since it has some vulnerabilities. >> But >>>>> this library was renamed to *org.apache.httpcomponents:httpclient* and >>>>> dependency to its latest version was added to the project, so it should >>>> be >>>>> used everywhere instead of the old one. >>>>> >>>>> By the way, *org.elasticsearch.client:elasticsearch-rest-client* 7.6.2 >>>> has >>>>> a dependency on *org.apache.httpcomponents:httpclient*, but not >>>>> *commons-httpclient*, so it is strange that you have some issues with >> the >>>>> older library. >>>>> >>>>> Kind regards, >>>>> Volodymyr Vysotskyi >>>>> >>>>> >>>>> On Mon, Apr 20, 2020 at 4:11 AM Charles Givre <[email protected]> >> wrote: >>>>> >>>>>> Hello all, >>>>>> I am doing some work for an Elasticsearch (ES) storage plugin. I >>>> recently >>>>>> rebased on the current master and found that broke my plugin due to a >>>>>> dependency exclusion. Basically, the ES modules need the >>>>>> commons-httpclient which seems to be excluded by the HBase plugin. My >>>>>> question is can I "un-exclude" it or do something in Maven so that the >>>> code >>>>>> gets these dependencies? >>>>>> Thanks, >>>>>> -- C >>>> >>>> >>
