Hi all, Thanks, Charles, for sharing additional info. I have checked out to your branch, and now I see that elasticsearch-hadoop uses classes from the commons-httpclient library. Since ES is an open-source project, could you please create an issue for them to update this library? For now, I agree with Paul regarding retaining the older version.
When I've excluded commons-httpclient, I had a similar issue with nl.basjes.parse.httpdlog:httpdlog-parser, but after I have reported an issue ticket regarding updating this library, Niels rapidly fixed it and released a new version (thanks a lot to Niels for this!). Kind regards, Volodymyr Vysotskyi On Fri, Apr 24, 2020 at 5:58 AM Paul Rogers <[email protected]> wrote: > Hi All, > I think there may be a bit of confusion. It may be true that some of > Drill's dependencies now use the newer version of the library > httpcomponents:httpclient. However, it looks like ES directly depends on > the older flavor. > > We have pom file entries which exclude that old version. As a result, ES > can't find the older version at run time. > > So, maybe two choices. 1) convince ES to use the newer version (how?), or > 2 retain the older version (don't exclude it.) > > Would sure be nice if each storage plugin could run in its own class > loader to avoid these issues. We're slowly moving in that direction. > > Thanks, > - Paul > > > > On Thursday, April 23, 2020, 5:35:55 PM PDT, Charles Givre < > [email protected]> wrote: > > Hi Vova, > Thanks for the response. I've been slowly poking at a storage plugin for > ElasticSearch.[1] I was going to do some work on it, but after rebasing to > the latest master, I'm getting errors in my unit tests that were not there > before. > > Here's the relevant snippet of the dependency tree: > > [INFO] +- > org.elasticsearch.client:elasticsearch-rest-client:jar:7.6.2:compile > [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile > [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile > [INFO] | \- org.apache.httpcomponents:httpcore-nio:jar:4.4.12:compile > [INFO] +- org.elasticsearch:elasticsearch-hadoop:jar:7.6.2:compile > > Currently, I don't have the dependency excluded or anything like that in > the pom.xml for the storage plugin so I would assume that the dependency > would be included, but it doesn't seem to be. Would you have any > suggestions as to how to fix it? > > > Thanks, > -- C > > > > Here's the full stack trace: > 19:58:51.940 [Time-limited test] DEBUG o.a.d.e.s.e.TestElasticQueries - > select * from elasticsearch.employee.`developer` > 19:58:52.011 [215dd444-5b82-78b6-6222-25db75b0a934:foreman] DEBUG > o.a.d.e.s.e.ElasticSearchGroupScan - Getting region locations > > org.apache.drill.exec.rpc.RpcException: > org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: > ClassNotFoundException: > org.apache.commons.httpclient.protocol.ProtocolSocketFactory > > > Please, refer to logs for more information. > > [Error Id: f43ab4ec-d557-45df-a41d-1a0221c3ffdb on 192.168.1.25:31013] > > at > org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:59) > at > org.apache.drill.exec.client.DrillClient$ListHoldingResultsListener.getResults(DrillClient.java:881) > at > org.apache.drill.exec.client.DrillClient.runQuery(DrillClient.java:583) > at org.apache.drill.test.QueryBuilder.results(QueryBuilder.java:331) > at > org.apache.drill.test.ClusterFixture$FixtureTestServices.testRunAndReturn(ClusterFixture.java:615) > at > org.apache.drill.test.DrillTestWrapper.testRunAndReturn(DrillTestWrapper.java:938) > at > org.apache.drill.test.DrillTestWrapper.compareUnorderedResults(DrillTestWrapper.java:533) > at > org.apache.drill.test.DrillTestWrapper.run(DrillTestWrapper.java:172) > at org.apache.drill.test.TestBuilder.go(TestBuilder.java:145) > at > org.apache.drill.exec.store.elasticsearch.TestElasticQueries.testSimpleStarQuery(TestElasticQueries.java:83) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.drill.common.exceptions.UserRemoteException: SYSTEM > ERROR: ClassNotFoundException: > org.apache.commons.httpclient.protocol.ProtocolSocketFactory > > > Please, refer to logs for more information. > > [Error Id: f43ab4ec-d557-45df-a41d-1a0221c3ffdb on 192.168.1.25:31013] > at > org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:125) > at > org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422) > at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) > at > org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) > at > io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) > at > io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) > at > io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) > at > io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) > at > io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > ... 1 more > Caused by: org.apache.drill.exec.work.foreman.ForemanException: > Unexpected exception during fragment initialization: > org/apache/commons/httpclient/protocol/ProtocolSocketFactory > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:301) > at .......(:0) > Caused by: java.lang.Error: > org/apache/commons/httpclient/protocol/ProtocolSocketFactory > at > org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory.create(CommonsHttpTransportFactory.java:40) > at > org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:99) > at > org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:82) > at > org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:58) > at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:101) > at > org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:327) > at > org.elasticsearch.hadoop.rest.RestService.findPartitions(RestService.java:220) > at > org.apache.drill.exec.store.elasticsearch.ElasticSearchGroupScan.init(ElasticSearchGroupScan.java:196) > at > org.apache.drill.exec.store.elasticsearch.ElasticSearchGroupScan.<init>(ElasticSearchGroupScan.java:124) > at > org.apache.drill.exec.store.elasticsearch.ElasticSearchStoragePlugin.getPhysicalScan(ElasticSearchStoragePlugin.java:122) > at > org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:112) > at > org.apache.drill.exec.store.AbstractStoragePlugin.getPhysicalScan(AbstractStoragePlugin.java:117) > at > org.apache.drill.exec.planner.logical.DrillTable.getGroupScan(DrillTable.java:119) > at > org.apache.drill.exec.planner.common.DrillScanRelBase.<init>(DrillScanRelBase.java:51) > at > org.apache.drill.exec.planner.logical.DrillScanRel.<init>(DrillScanRel.java:79) > at > org.apache.drill.exec.planner.logical.DrillScanRel.<init>(DrillScanRel.java:66) > at > org.apache.drill.exec.planner.logical.DrillScanRel.<init>(DrillScanRel.java:59) > at > org.apache.drill.exec.planner.logical.DrillScanRule.onMatch(DrillScanRule.java:38) > at > org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:208) > at > org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:633) > at > org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:327) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:405) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:351) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToRawDrel(DefaultSqlHandler.java:245) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:308) > at > org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.getPlan(DefaultSqlHandler.java:173) > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:283) > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPhysicalPlan(DrillSqlWorker.java:163) > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.convertPlan(DrillSqlWorker.java:128) > at > org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:93) > at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:593) > at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:274) > ... 1 more > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.httpclient.protocol.ProtocolSocketFactory > at .......(:0) > ... 33 more > > > Here is the dependency tree: > > (base) ➜ storage-elastic git:(storage-elastic2) ✗ mvn dependency:tree > [INFO] Scanning for projects... > [INFO] > ------------------------------------------------------------------------ > [INFO] Detecting the operating system and CPU architecture > [INFO] > ------------------------------------------------------------------------ > [INFO] os.detected.name: osx > [INFO] os.detected.arch: x86_64 > [INFO] os.detected.version: 10.15 > [INFO] os.detected.version.major: 10 > [INFO] os.detected.version.minor: 15 > [INFO] os.detected.classifier: osx-x86_64 > [INFO] > [INFO] --------< org.apache.drill.contrib:drill-storage-elasticsearch > >-------- > [INFO] Building contrib/storage-elasticsearch 1.18.0-SNAPSHOT > [INFO] --------------------------------[ jar > ]--------------------------------- > [INFO] > [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ > drill-storage-elasticsearch --- > [INFO] > org.apache.drill.contrib:drill-storage-elasticsearch:jar:1.18.0-SNAPSHOT > [INFO] +- org.apache.drill.exec:drill-java-exec:jar:1.18.0-SNAPSHOT:compile > [INFO] | +- org.apache.httpcomponents:httpasyncclient:jar:4.1.4:compile > [INFO] | +- org.ow2.asm:asm-commons:jar:7.3.1:compile > [INFO] | | +- org.ow2.asm:asm:jar:7.3.1:compile > [INFO] | | +- org.ow2.asm:asm-tree:jar:7.3.1:compile > [INFO] | | \- org.ow2.asm:asm-analysis:jar:7.3.1:compile > [INFO] | +- org.ow2.asm:asm-util:jar:7.3.1:compile > [INFO] | +- org.apache.commons:commons-pool2:jar:2.6.0:compile > [INFO] | +- com.univocity:univocity-parsers:jar:2.8.3:compile > [INFO] | +- org.apache.commons:commons-math:jar:2.2:compile > [INFO] | +- com.thoughtworks.paranamer:paranamer:jar:2.5.6:compile > [INFO] | +- xerces:xercesImpl:jar:2.12.0:compile > [INFO] | | \- xml-apis:xml-apis:jar:1.4.01:compile > [INFO] | +- xalan:xalan:jar:2.7.2:compile > [INFO] | | \- xalan:serializer:jar:2.7.2:compile > [INFO] | +- com.sun.codemodel:codemodel:jar:2.6:compile > [INFO] | +- org.eclipse.jetty:jetty-server:jar:9.3.28.v20191105:compile > [INFO] | | +- javax.servlet:javax.servlet-api:jar:3.1.0:compile > [INFO] | | +- org.eclipse.jetty:jetty-http:jar:9.3.28.v20191105:compile > [INFO] | | \- org.eclipse.jetty:jetty-io:jar:9.3.28.v20191105:compile > [INFO] | +- org.eclipse.jetty:jetty-servlet:jar:9.3.28.v20191105:compile > [INFO] | | \- > org.eclipse.jetty:jetty-security:jar:9.3.28.v20191105:compile > [INFO] | +- org.eclipse.jetty:jetty-servlets:jar:9.3.28.v20191105:compile > [INFO] | +- > org.glassfish.jersey.containers:jersey-container-jetty-servlet:jar:2.25.1:compile > [INFO] | | +- > org.glassfish.jersey.containers:jersey-container-servlet:jar:2.25.1:compile > [INFO] | | | \- > org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.25.1:compile > [INFO] | | +- > org.glassfish.jersey.containers:jersey-container-jetty-http:jar:2.25.1:compile > [INFO] | | | +- > org.glassfish.hk2.external:javax.inject:jar:2.5.0-b32:compile > [INFO] | | | \- > org.eclipse.jetty:jetty-continuation:jar:9.2.14.v20151106:compile > [INFO] | | +- org.eclipse.jetty:jetty-webapp:jar:9.3.28.v20191105:compile > [INFO] | | | \- org.eclipse.jetty:jetty-xml:jar:9.3.28.v20191105:compile > [INFO] | | +- org.glassfish.jersey.core:jersey-common:jar:2.25.1:compile > [INFO] | | | +- > org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.25.1:compile > [INFO] | | | +- org.glassfish.hk2:hk2-api:jar:2.5.0-b32:compile > [INFO] | | | | +- org.glassfish.hk2:hk2-utils:jar:2.5.0-b32:compile > [INFO] | | | | \- > org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b32:compile > [INFO] | | | +- org.glassfish.hk2:hk2-locator:jar:2.5.0-b32:compile > [INFO] | | | \- > org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile > [INFO] | | +- org.glassfish.jersey.core:jersey-server:jar:2.25.1:compile > [INFO] | | | +- > org.glassfish.jersey.core:jersey-client:jar:2.25.1:compile > [INFO] | | | +- > org.glassfish.jersey.media:jersey-media-jaxb:jar:2.25.1:compile > [INFO] | | | \- javax.validation:validation-api:jar:1.1.0.Final:compile > [INFO] | | \- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile > [INFO] | +- > org.glassfish.jersey.media:jersey-media-multipart:jar:2.25.1:compile > [INFO] | | \- org.jvnet.mimepull:mimepull:jar:1.9.6:compile > [INFO] | +- > org.glassfish.jersey.media:jersey-media-json-jackson:jar:2.25.1:compile > [INFO] | | +- > org.glassfish.jersey.ext:jersey-entity-filtering:jar:2.25.1:compile > [INFO] | | \- > com.fasterxml.jackson.jaxrs:jackson-jaxrs-base:jar:2.8.4:compile > [INFO] | +- > com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider:jar:2.10.3:compile > [INFO] | | \- > com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.10.3:compile > [INFO] | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile > [INFO] | | \- > jakarta.activation:jakarta.activation-api:jar:1.2.1:compile > [INFO] | +- org.mongodb:mongo-java-driver:jar:3.8.0:compile > [INFO] | +- > com.fasterxml.jackson.module:jackson-module-afterburner:jar:2.10.3:compile > [INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.10.3:compile > [INFO] | +- > org.honton.chas.hocon:jackson-dataformat-hocon:jar:1.1.1:compile > [INFO] | +- > org.glassfish.jersey.ext:jersey-mvc-freemarker:jar:2.25.1:compile > [INFO] | | \- org.glassfish.jersey.ext:jersey-mvc:jar:2.25.1:compile > [INFO] | +- > com.github.vvysotskyi.drill-calcite:calcite-core:jar:1.21.0-drill-r0:compile > [INFO] | | +- > com.github.vvysotskyi.drill-calcite:calcite-linq4j:jar:1.21.0-drill-r0:compile > [INFO] | | +- org.apache.commons:commons-dbcp2:jar:2.6.0:compile > [INFO] | | +- com.esri.geometry:esri-geometry-api:jar:2.2.0:compile > [INFO] | | +- > com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.9.9:compile > [INFO] | | | \- org.yaml:snakeyaml:jar:1.26:compile > [INFO] | | +- com.google.guava:guava:jar:19.0:compile > [INFO] | | +- com.yahoo.datasketches:sketches-core:jar:0.9.0:compile > [INFO] | | | \- com.yahoo.datasketches:memory:jar:0.9.0:compile > [INFO] | | +- com.jayway.jsonpath:json-path:jar:2.4.0:compile > [INFO] | | \- net.hydromatic:aggdesigner-algorithm:jar:6.0:compile > [INFO] | | \- commons-lang:commons-lang:jar:2.4:compile > [INFO] | +- org.apache.calcite.avatica:avatica:jar:1.15.0:compile > [INFO] | +- net.sf.jpam:jpam:jar:1.1:compile > [INFO] | +- org.bouncycastle:bcpkix-jdk15on:jar:1.60:compile > [INFO] | | \- org.bouncycastle:bcprov-jdk15on:jar:1.60:compile > [INFO] | +- org.freemarker:freemarker:jar:2.3.28:compile > [INFO] | +- org.apache.parquet:parquet-column:jar:1.11.0:compile > [INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.11.0:compile > [INFO] | +- org.apache.parquet:parquet-hadoop:jar:1.11.0:compile > [INFO] | | +- > org.apache.parquet:parquet-format-structures:jar:1.11.0:compile > [INFO] | | +- org.apache.parquet:parquet-jackson:jar:1.11.0:compile > [INFO] | | \- commons-pool:commons-pool:jar:1.6:compile > [INFO] | +- org.apache.parquet:parquet-format:jar:2.8.0:compile > [INFO] | | \- javax.annotation:javax.annotation-api:jar:1.3.2:compile > [INFO] | +- org.apache.parquet:parquet-common:jar:1.11.0:compile > [INFO] | | \- org.apache.yetus:audience-annotations:jar:0.11.0:compile > [INFO] | +- javax.inject:javax.inject:jar:1:compile > [INFO] | +- org.apache.drill:drill-protocol:jar:1.18.0-SNAPSHOT:compile > [INFO] | | +- io.protostuff:protostuff-core:jar:1.7.1:compile > [INFO] | | | \- io.protostuff:protostuff-api:jar:1.7.1:compile > [INFO] | | \- io.protostuff:protostuff-json:jar:1.7.1:compile > [INFO] | +- org.apache.drill:drill-common:jar:1.18.0-SNAPSHOT:compile > [INFO] | +- org.apache.drill:drill-logical:jar:1.18.0-SNAPSHOT:compile > [INFO] | | \- org.antlr:antlr4-runtime:jar:4.8-1:compile > [INFO] | +- org.apache.drill.exec:drill-rpc:jar:1.18.0-SNAPSHOT:compile > [INFO] | | \- > io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.0.48.Final:compile > [INFO] | +- > org.apache.drill.memory:drill-memory-base:jar:1.18.0-SNAPSHOT:compile > [INFO] | +- org.apache.drill.exec:vector:jar:1.18.0-SNAPSHOT:compile > [INFO] | +- > org.apache.drill.metastore:drill-metastore-api:jar:1.18.0-SNAPSHOT:compile > [INFO] | | \- > com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.10.3:compile > [INFO] | +- com.beust:jcommander:jar:1.30:compile > [INFO] | +- org.apache.curator:curator-client:jar:4.3.0:compile > [INFO] | | \- org.apache.zookeeper:zookeeper:jar:3.5.7:compile > [INFO] | | +- org.apache.zookeeper:zookeeper-jute:jar:3.5.7:compile > [INFO] | | \- > io.netty:netty-transport-native-epoll:jar:4.1.45.Final:compile > [INFO] | | \- > io.netty:netty-transport-native-unix-common:jar:4.1.45.Final:compile > [INFO] | +- org.apache.curator:curator-framework:jar:4.3.0:compile > [INFO] | +- org.apache.curator:curator-recipes:jar:4.3.0:compile > [INFO] | +- org.apache.curator:curator-x-discovery:jar:4.3.0:compile > [INFO] | +- com.carrotsearch:hppc:jar:0.7.1:compile > [INFO] | +- com.google.protobuf:protobuf-java:jar:3.11.1:compile > [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile > [INFO] | +- joda-time:joda-time:jar:2.10.5:compile > [INFO] | +- org.codehaus.janino:janino:jar:3.0.11:compile > [INFO] | +- org.codehaus.janino:commons-compiler:jar:3.0.11:compile > [INFO] | +- com.clearspring.analytics:stream:jar:2.7.0:compile > [INFO] | | \- it.unimi.dsi:fastutil:jar:6.5.7:compile > [INFO] | +- com.tdunning:t-digest:jar:3.2:compile > [INFO] | +- org.apache.hadoop:hadoop-common:jar:3.2.1:compile > [INFO] | | +- org.apache.hadoop:hadoop-annotations:jar:3.2.1:compile > [INFO] | | +- commons-cli:commons-cli:jar:1.4:compile > [INFO] | | +- org.apache.commons:commons-math3:jar:3.1.1:compile > [INFO] | | +- commons-net:commons-net:jar:3.6:compile > [INFO] | | +- commons-collections:commons-collections:jar:3.2.2:compile > [INFO] | | +- org.eclipse.jetty:jetty-util:jar:9.3.28.v20191105:compile > [INFO] | | +- javax.servlet.jsp:jsp-api:jar:2.1:runtime > [INFO] | | +- com.sun.jersey:jersey-servlet:jar:1.19:compile > [INFO] | | +- commons-beanutils:commons-beanutils:jar:1.9.4:compile > [INFO] | | +- org.apache.commons:commons-configuration2:jar:2.1.1:compile > [INFO] | | +- org.apache.commons:commons-text:jar:1.6:compile > [INFO] | | +- com.google.re2j:re2j:jar:1.1:compile > [INFO] | | +- com.google.code.gson:gson:jar:2.2.4:compile > [INFO] | | +- org.apache.hadoop:hadoop-auth:jar:3.2.1:compile > [INFO] | | +- com.jcraft:jsch:jar:0.1.54:compile > [INFO] | | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:compile > [INFO] | | +- org.apache.commons:commons-compress:jar:1.20:compile > [INFO] | | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile > [INFO] | | +- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:compile > [INFO] | | \- dnsjava:dnsjava:jar:2.1.7:compile > [INFO] | +- org.apache.hadoop:hadoop-client:jar:3.2.1:compile > [INFO] | | +- org.apache.hadoop:hadoop-hdfs-client:jar:3.2.1:compile > [INFO] | | | \- com.squareup.okhttp:okhttp:jar:2.7.5:compile > [INFO] | | | \- com.squareup.okio:okio:jar:1.6.0:compile > [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:3.2.1:compile > [INFO] | | +- > org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.2.1:compile > [INFO] | | | \- org.apache.hadoop:hadoop-yarn-common:jar:3.2.1:compile > [INFO] | | | \- javax.xml.bind:jaxb-api:jar:2.2.11:compile > [INFO] | | \- > org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:3.2.1:compile > [INFO] | | \- > org.apache.hadoop:hadoop-mapreduce-client-common:jar:3.2.1:compile > [INFO] | +- org.apache.avro:avro:jar:1.9.1:compile > [INFO] | +- org.apache.avro:avro-mapred:jar:1.9.1:compile > [INFO] | | +- org.apache.avro:avro-ipc:jar:1.9.1:compile > [INFO] | | | \- org.apache.velocity:velocity-engine-core:jar:2.0:compile > [INFO] | | \- org.apache.avro:avro-ipc-jetty:jar:1.9.1:compile > [INFO] | +- nl.basjes.parse.httpdlog:httpdlog-parser:jar:5.3:compile > [INFO] | | +- nl.basjes.parse:parser-core:jar:5.3:compile > [INFO] | | \- com.maxmind.geoip2:geoip2:jar:2.13.1:compile > [INFO] | | \- com.maxmind.db:maxmind-db:jar:1.3.1:compile > [INFO] | +- org.kohsuke:libpam4j:jar:1.8-rev2:compile > [INFO] | | \- net.java.dev.jna:jna:jar:4.1.0:compile > [INFO] | +- io.netty:netty-tcnative:jar:2.0.1.Final:runtime (optional) > [INFO] | +- com.drewnoakes:metadata-extractor:jar:2.13.0:compile > [INFO] | | \- com.adobe.xmp:xmpcore:jar:6.0.6:compile > [INFO] | +- fr.bmartel:pcapngdecoder:jar:1.2:compile > [INFO] | \- sqlline:sqlline:jar:1.9.0:compile > [INFO] | +- org.jline:jline-terminal:jar:3.12.1:compile > [INFO] | +- org.jline:jline-reader:jar:3.12.1:compile > [INFO] | +- org.jline:jline-terminal-jansi:jar:3.12.1:compile > [INFO] | | \- org.fusesource.jansi:jansi:jar:1.18:compile > [INFO] | +- org.jline:jline-terminal-jna:jar:3.12.1:compile > [INFO] | \- org.jline:jline-builtins:jar:3.12.1:compile > [INFO] | \- org.jline:jline-style:jar:3.12.1:compile > [INFO] +- > org.elasticsearch.client:elasticsearch-rest-client:jar:7.6.2:compile > [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile > [INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.12:compile > [INFO] | \- org.apache.httpcomponents:httpcore-nio:jar:4.4.12:compile > [INFO] +- org.elasticsearch:elasticsearch-hadoop:jar:7.6.2:compile > [INFO] +- org.elasticsearch.client:transport:jar:7.6.2:compile > [INFO] | +- org.elasticsearch:elasticsearch:jar:7.6.2:compile > [INFO] | | +- org.elasticsearch:elasticsearch-core:jar:7.6.2:compile > [INFO] | | +- org.elasticsearch:elasticsearch-secure-sm:jar:7.6.2:compile > [INFO] | | +- org.elasticsearch:elasticsearch-x-content:jar:7.6.2:compile > [INFO] | | | +- > com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.8.11:compile > [INFO] | | | \- > com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.8.11:compile > [INFO] | | +- org.elasticsearch:elasticsearch-geo:jar:7.6.2:compile > [INFO] | | +- org.apache.lucene:lucene-core:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-analyzers-common:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-backward-codecs:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-grouping:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-highlighter:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-join:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-memory:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-misc:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-queries:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-queryparser:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-sandbox:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-spatial:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-spatial-extras:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-spatial3d:jar:8.4.0:compile > [INFO] | | +- org.apache.lucene:lucene-suggest:jar:8.4.0:compile > [INFO] | | +- org.elasticsearch:elasticsearch-cli:jar:7.6.2:compile > [INFO] | | | \- net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile > [INFO] | | +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile > [INFO] | | +- org.apache.logging.log4j:log4j-api:jar:2.11.1:compile > [INFO] | | \- org.elasticsearch:jna:jar:4.5.1:compile > [INFO] | +- > org.elasticsearch.plugin:transport-netty4-client:jar:7.6.2:compile > [INFO] | | +- io.netty:netty-codec-http:jar:4.1.43.Final:compile > [INFO] | | \- io.netty:netty-resolver:jar:4.1.43.Final:compile > [INFO] | +- org.elasticsearch.plugin:reindex-client:jar:7.6.2:compile > [INFO] | | \- > org.elasticsearch:elasticsearch-ssl-config:jar:7.6.2:compile > [INFO] | +- > org.elasticsearch.plugin:lang-mustache-client:jar:7.6.2:compile > [INFO] | | \- > com.github.spullara.mustache.java:compiler:jar:0.9.6:compile > [INFO] | +- org.elasticsearch.plugin:percolator-client:jar:7.6.2:compile > [INFO] | +- org.elasticsearch.plugin:parent-join-client:jar:7.6.2:compile > [INFO] | \- org.elasticsearch.plugin:rank-eval-client:jar:7.6.2:compile > [INFO] +- > org.apache.drill.exec:drill-java-exec:jar:tests:1.18.0-SNAPSHOT:test > [INFO] +- org.apache.drill:drill-common:jar:tests:1.18.0-SNAPSHOT:test > [INFO] | +- com.typesafe:config:jar:1.0.0:compile > [INFO] | +- org.apache.commons:commons-lang3:jar:3.10:compile > [INFO] | +- org.msgpack:msgpack:jar:0.6.6:compile > [INFO] | | \- com.googlecode.json-simple:json-simple:jar:1.1.1:compile > [INFO] | +- org.reflections:reflections:jar:0.9.10:compile > [INFO] | | \- com.google.code.findbugs:annotations:jar:2.0.1:compile > [INFO] | +- org.javassist:javassist:jar:3.27.0-GA:compile > [INFO] | +- > com.fasterxml.jackson.core:jackson-annotations:jar:2.10.3:compile > [INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.10.3:compile > [INFO] | +- io.dropwizard.metrics:metrics-core:jar:4.0.2:compile > [INFO] | +- io.dropwizard.metrics:metrics-servlets:jar:4.0.2:compile > [INFO] | | +- > io.dropwizard.metrics:metrics-healthchecks:jar:4.0.2:compile > [INFO] | | +- io.dropwizard.metrics:metrics-json:jar:4.0.2:compile > [INFO] | | \- com.papertrail:profiler:jar:1.0.2:compile > [INFO] | +- io.dropwizard.metrics:metrics-jvm:jar:4.0.2:compile > [INFO] | \- io.dropwizard.metrics:metrics-jmx:jar:4.0.2:compile > [INFO] +- commons-codec:commons-codec:jar:1.14:compile > [INFO] +- io.netty:netty-handler:jar:4.0.48.Final:compile > [INFO] | +- io.netty:netty-buffer:jar:4.0.48.Final:compile > [INFO] | +- io.netty:netty-transport:jar:4.0.48.Final:compile > [INFO] | \- io.netty:netty-codec:jar:4.0.48.Final:compile > [INFO] +- io.netty:netty-common:jar:4.0.48.Final:compile > [INFO] +- org.apache.drill:drill-shaded-guava:jar:28.2-jre:compile > [INFO] | +- > com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile > [INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile > [INFO] | +- org.checkerframework:checker-qual:jar:2.10.0:compile > [INFO] | +- > com.google.errorprone:error_prone_annotations:jar:2.3.4:compile > [INFO] | \- com.google.j2objc:j2objc-annotations:jar:1.3:compile > [INFO] +- org.slf4j:slf4j-api:jar:1.7.26:compile > [INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.26:compile > [INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.26:compile > [INFO] +- org.slf4j:log4j-over-slf4j:jar:1.7.26:compile > [INFO] +- commons-io:commons-io:jar:2.4:compile > [INFO] +- org.jmockit:jmockit:jar:1.47:test > [INFO] +- junit:junit:jar:4.12:test > [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test > [INFO] +- org.mockito:mockito-core:jar:2.23.4:test > [INFO] | +- net.bytebuddy:byte-buddy:jar:1.9.3:test > [INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.9.3:test > [INFO] | \- org.objenesis:objenesis:jar:2.6:test > [INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:test > [INFO] +- ch.qos.logback:logback-core:jar:1.2.3:test > [INFO] +- > de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic:jar:0.9.44:test > [INFO] | +- de.huxhorn.sulky:de.huxhorn.sulky.codec:jar:0.9.17:test > [INFO] | | \- de.huxhorn.sulky:de.huxhorn.sulky.io:jar:0.9.17:test > [INFO] | +- > de.huxhorn.lilith:de.huxhorn.lilith.logback.converter-classic:jar:0.9.44:test > [INFO] | | +- > de.huxhorn.lilith:de.huxhorn.lilith.data.converter:jar:0.9.44:test > [INFO] | | +- > de.huxhorn.lilith:de.huxhorn.lilith.logback.classic:jar:0.9.44:test > [INFO] | | \- > de.huxhorn.lilith:de.huxhorn.lilith.data.logging:jar:0.9.44:test > [INFO] | | +- > de.huxhorn.sulky:de.huxhorn.sulky.formatting:jar:0.9.17:test > [INFO] | | \- > de.huxhorn.lilith:de.huxhorn.lilith.data.eventsource:jar:0.9.44:test > [INFO] | +- > de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-core:jar:0.9.44:test > [INFO] | | \- de.huxhorn.lilith:de.huxhorn.lilith.sender:jar:0.9.44:test > [INFO] | \- > de.huxhorn.lilith:de.huxhorn.lilith.data.logging.protobuf:jar:0.9.44:test > [INFO] +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile > [INFO] +- org.apache.kerby:kerb-client:jar:1.0.0:test > [INFO] | +- org.apache.kerby:kerby-config:jar:1.0.0:test > [INFO] | +- org.apache.kerby:kerb-common:jar:1.0.0:test > [INFO] | | \- org.apache.kerby:kerb-crypto:jar:1.0.0:test > [INFO] | \- org.apache.kerby:kerb-util:jar:1.0.0:test > [INFO] +- org.apache.kerby:kerb-core:jar:1.0.0:test > [INFO] | \- org.apache.kerby:kerby-pkix:jar:1.0.0:test > [INFO] | +- org.apache.kerby:kerby-asn1:jar:1.0.0:test > [INFO] | \- org.apache.kerby:kerby-util:jar:1.0.0:test > [INFO] \- org.apache.kerby:kerb-simplekdc:jar:1.0.0:test > [INFO] \- org.apache.kerby:kerb-admin:jar:1.0.0:test > [INFO] +- org.apache.kerby:kerb-server:jar:1.0.0:test > [INFO] | \- org.apache.kerby:kerb-identity:jar:1.0.0:test > [INFO] \- org.apache.kerby:kerby-xdr:jar:1.0.0:test > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 1.402 s > [INFO] Finished at: 2020-04-23T20:30:50-04:00 > [INFO] > ------------------------------------------------------------------------ > (base) ➜ storage-elastic git:(storage-elastic2) ✗ > > > > > > > [1]: https://github.com/apache/drill/pull/1961 < > https://github.com/apache/drill/pull/1961> > > > > > > On Apr 22, 2020, at 11:11 AM, Vova Vysotskyi <[email protected]> wrote: > > > > Hi Charles, > > > > Could you please share more details about your problem, for example, > which > > one dependency uses commons-httpclient, which errors you see, etc. > > > > Kind regards, > > Volodymyr Vysotskyi > > > > > > On Wed, Apr 22, 2020 at 5:23 PM Charles Givre <[email protected]> wrote: > > > >> Hi Volodymyr, > >> Thanks for the response. Do you have any suggestions as to how to fix > >> this? I don't really know enough Maven to fix this. I've tried > manually > >> including the library in the pom, removing exclusions etc. > >> Thanks, > >> -- C > >> > >> > >> > >>> On Apr 20, 2020, at 4:15 AM, Vova Vysotskyi <[email protected]> wrote: > >>> > >>> Hi Charles, > >>> > >>> *commons-httpclient* was excluded since it has some vulnerabilities. > But > >>> this library was renamed to *org.apache.httpcomponents:httpclient* and > >>> dependency to its latest version was added to the project, so it should > >> be > >>> used everywhere instead of the old one. > >>> > >>> By the way, *org.elasticsearch.client:elasticsearch-rest-client* 7.6.2 > >> has > >>> a dependency on *org.apache.httpcomponents:httpclient*, but not > >>> *commons-httpclient*, so it is strange that you have some issues with > the > >>> older library. > >>> > >>> Kind regards, > >>> Volodymyr Vysotskyi > >>> > >>> > >>> On Mon, Apr 20, 2020 at 4:11 AM Charles Givre <[email protected]> > wrote: > >>> > >>>> Hello all, > >>>> I am doing some work for an Elasticsearch (ES) storage plugin. I > >> recently > >>>> rebased on the current master and found that broke my plugin due to a > >>>> dependency exclusion. Basically, the ES modules need the > >>>> commons-httpclient which seems to be excluded by the HBase plugin. My > >>>> question is can I "un-exclude" it or do something in Maven so that the > >> code > >>>> gets these dependencies? > >>>> Thanks, > >>>> -- C > >> > >> >
