Thanks, Gonzalo! That actually got me further. A Solr exception later (got around it by removing 4.10 Solr libs which I assume don't work with my 5.3 for which I already have libs in CP), then I got another error pointing the same sort of issue.
So I just went ahead and removed all duplicated jars (though many were din different versions) from kite. With something as ugly as this: for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls /opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm $lib*; fi; done Then Flume complained about some codahale metrics class missing, so I moved back the metrics-core library to kite. And now I get this: java.lang.VerifyError: Cannot inherit from final class I'm not sure if you need the whole stacktrace (I will paste it below) but I have again the suspicion of duplicated stuff or mismatched versions. I'm not sure I'm on the right path, so you were pointing me to different ones: - add the integration with Solr. Can you tell me which are the jars needed? >From which subproject? - look at the version of Flume distributed by Cloudera. I can't find just Flume, I suppose I need to get the whole CDH or something like that? Thanks again for all your help! Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gherre...@gmail.com> wrote: > That's it. > Remove that jar from the Flume classpath so it doesn't interfere with the > one in Flume (version 2.5) > > Note Flume already has the kite libraries. I would just add the > integration with Solr (I suggest you have a look at the version of Flume > distributed by Cloudera) > > Gonzalo > > > On 17 September 2015 at 12:39, Radu Gheorghe <radu.gheor...@sematext.com> > wrote: > >> Hi Gonzalo, >> >> Thanks for your reply. Unfortunately, I've already went that route with >> no success. I can only find out one jar holding that, located in Morphlines: >> >> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all> >> grep -RHls javax.servlet.AsyncContext . >> ./target/lib/javax.servlet-3.0.0.v201112011016.jar >> >> grepping in the Solr and Flume directories got me no result. Do you have >> more ideas? >> >> Best regards, >> Radu >> >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gherre...@gmail.com> >> wrote: >> >>> Usually that means you are loading different version of the servlet API. >>> You need to do a bit of classpath troubleshooting to find which jars >>> contain javax.servlet.AsyncContext and keep just the one included in >>> Flume/lib >>> >>> Regards, >>> Gonzalo >>> >>> On 17 September 2015 at 11:31, Radu Gheorghe <radu.gheor...@sematext.com >>> > wrote: >>> >>>> Hello flume users! >>>> >>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I did >>>> that with 1.5 but now I seem to fail :( >>>> >>>> I got this error in the log (full exception below): >>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s >>>> signer information does not match signer information of other classes in >>>> the same package >>>> >>>> Basically, what I did was to install Flume 1.6, install Solr 5.3, >>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to >>>> Flume's classpath. Here's the full command line: >>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m >>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp >>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*' >>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f >>>> conf/flume-conf.properties >>>> >>>> Here's my Java version, if it matters: >>>> java version "1.7.0_79" >>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64) >>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) >>>> >>>> From what I can understand out off the Internet (thanks, Internet!) >>>> this may be caused by multiple jars having different versions of the same >>>> thing. But I can only find it in Morphlines (via grep -RHls in all the >>>> classpaths). Maybe I'm bumping into >>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to >>>> verify this by compiling Kite 0.12.0 but it failed with: >>>> [ERROR] Failed to execute goal >>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project kite-data: >>>> Too many files with unapproved license: 2 See RAT report.... >>>> >>>> So then I compiled only morphlines and that worked, only to bump into >>>> the same error. >>>> >>>> As promised, below is the full exception. For completeness, I'm also >>>> including the main config and the morphline config: >>>> >>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8] >>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) - >>>> Unable to start SinkRunner: { >>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 >>>> counterGroup:{ name:null counters:{} } } - Exception follows. >>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s >>>> signer information does not match signer information of other classes in >>>> the same package >>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) >>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) >>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794) >>>> at >>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800) >>>> at >>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>> at >>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224) >>>> at >>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176) >>>> at >>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91) >>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43) >>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40) >>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126) >>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55) >>>> at >>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101) >>>> at >>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97) >>>> at >>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) >>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79) >>>> at >>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> >>>> > cat conf/flume-conf.properties >>>> agent.sources = spoolDir >>>> agent.channels = spillableMem >>>> agent.sinks = solr >>>> >>>> agent.sources.spoolDir.type = spooldir >>>> agent.sources.spoolDir.spoolDir = /opt/logs >>>> agent.sources.spoolDir.batchSize = 1000 >>>> agent.sources.spoolDir.channels = spillableMem >>>> >>>> agent.channels.spillableMem.type = SPILLABLEMEMORY >>>> agent.channels.spillableMem.memoryCapacity = 100000 >>>> agent.channels.spillableMem.overflowCapacity = 1000000 >>>> agent.channels.spillableMem.checkpointDir = /var/run/flume >>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data >>>> >>>> agent.sinks.solr.type = >>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink >>>> agent.sinks.solr.channel = spillableMem >>>> agent.sinks.solr.morphlineFile = conf/morphline.conf >>>> agent.sinks.solr.morphlineId = 1 >>>> agent.sinks.solr.batchSize = 10000 >>>> >>>> > cat conf/morphline.conf >>>> SOLR_LOCATOR : { >>>> collection : gettingstarted >>>> solrUrl : "http://192.168.1.3:8983/solr/"; >>>> batchSize : 5000 >>>> } >>>> >>>> morphlines : [ >>>> { id : 1 >>>> commands : [ >>>> { readLine { charset : UTF-8 } } >>>> { >>>> grok { >>>> dictionaryFiles : [conf/grok-patterns] >>>> expressions : { >>>> message : """%{COMBINEDAPACHELOG}""" >>>> } >>>> } >>>> } >>>> # { generateUUID { >>>> # field : id >>>> # } >>>> # } >>>> { >>>> loadSolr { >>>> solrLocator : ${SOLR_LOCATOR} >>>> } >>>> } >>>> ] >>>> } >>>> ] >>>> >>>> I also have a grok-patterns file there, but I don't think it's relevant. >>>> >>>> Any ideas are much appreciated! >>>> >>>> Thanks and best regards, >>>> Radu >>>> -- >>>> Performance Monitoring * Log Analytics * Search Analytics >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>> >>> >> >
