Thanks, Gonzalo, for all your replies. I'm pretty sure the solution is within reach, I'm just running out of time. I did build flume from the repo you indicated but again it fails to start with some missing dependencies. I guess that indeed I should look at what are the required dependencies and provide them and only them.
I will try that later on and come back if I have more issues. -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Thu, Sep 17, 2015 at 4:28 PM, Gonzalo Herreros <gherre...@gmail.com> wrote: > Yes, it sounds like a dependency library version error. > You can take their the Cloudera Flume binaries, build from their branch of > Flume or just see how the manage the dependencies and do the same. > > https://github.com/cloudera/flume-ng (look at their poms) > You can download binaries here: > https://repository.cloudera.com/cloudera/cloudera-repos/ > > Gonzalo > > > On 17 September 2015 at 14:21, Radu Gheorghe <radu.gheor...@sematext.com> > wrote: > >> Thanks, Gonzalo! That actually got me further. A Solr exception later >> (got around it by removing 4.10 Solr libs which I assume don't work with my >> 5.3 for which I already have libs in CP), then I got another error pointing >> the same sort of issue. >> >> So I just went ahead and removed all duplicated jars (though many were >> din different versions) from kite. With something as ugly as this: >> for lib in `ls -1 | sed s/-[0-9].*.jar$//`; do ls >> /opt/apache-flume-1.6.0-bin/lib/$lib* 2>/dev/null; if [ $? -eq 0 ]; then rm >> $lib*; fi; done >> >> Then Flume complained about some codahale metrics class missing, so I >> moved back the metrics-core library to kite. And now I get this: >> >> java.lang.VerifyError: Cannot inherit from final class >> >> I'm not sure if you need the whole stacktrace (I will paste it below) but >> I have again the suspicion of duplicated stuff or mismatched versions. >> >> I'm not sure I'm on the right path, so you were pointing me to different >> ones: >> - add the integration with Solr. Can you tell me which are the jars >> needed? From which subproject? >> - look at the version of Flume distributed by Cloudera. I can't find just >> Flume, I suppose I need to get the whole CDH or something like that? >> >> Thanks again for all your help! >> >> Best regards, >> Radu >> >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> On Thu, Sep 17, 2015 at 2:50 PM, Gonzalo Herreros <gherre...@gmail.com> >> wrote: >> >>> That's it. >>> Remove that jar from the Flume classpath so it doesn't interfere with >>> the one in Flume (version 2.5) >>> >>> Note Flume already has the kite libraries. I would just add the >>> integration with Solr (I suggest you have a look at the version of Flume >>> distributed by Cloudera) >>> >>> Gonzalo >>> >>> >>> On 17 September 2015 at 12:39, Radu Gheorghe <radu.gheor...@sematext.com >>> > wrote: >>> >>>> Hi Gonzalo, >>>> >>>> Thanks for your reply. Unfortunately, I've already went that route with >>>> no success. I can only find out one jar holding that, located in >>>> Morphlines: >>>> >>>> radu@rgheorghe-suse:~/gits/kite/kite-morphlines/kite-morphlines-all> >>>> grep -RHls javax.servlet.AsyncContext . >>>> ./target/lib/javax.servlet-3.0.0.v201112011016.jar >>>> >>>> grepping in the Solr and Flume directories got me no result. Do you >>>> have more ideas? >>>> >>>> Best regards, >>>> Radu >>>> >>>> -- >>>> Performance Monitoring * Log Analytics * Search Analytics >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>>> On Thu, Sep 17, 2015 at 1:46 PM, Gonzalo Herreros <gherre...@gmail.com> >>>> wrote: >>>> >>>>> Usually that means you are loading different version of the servlet >>>>> API. >>>>> You need to do a bit of classpath troubleshooting to find which jars >>>>> contain javax.servlet.AsyncContext and keep just the one included in >>>>> Flume/lib >>>>> >>>>> Regards, >>>>> Gonzalo >>>>> >>>>> On 17 September 2015 at 11:31, Radu Gheorghe < >>>>> radu.gheor...@sematext.com> wrote: >>>>> >>>>>> Hello flume users! >>>>>> >>>>>> I'm trying to write log to Solr using Flume's MorphlineSolrSink. I >>>>>> did that with 1.5 but now I seem to fail :( >>>>>> >>>>>> I got this error in the log (full exception below): >>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s >>>>>> signer information does not match signer information of other classes in >>>>>> the same package >>>>>> >>>>>> Basically, what I did was to install Flume 1.6, install Solr 5.3, >>>>>> compile Kite SDK 1.1.0 and then added Solr and "morphlines-all" jars to >>>>>> Flume's classpath. Here's the full command line: >>>>>> exec /usr/lib64/jvm/java/bin/java -Xms1000m -Xmx2000m >>>>>> -Dcom.sun.management.jmxremote -Dmetadata-complete=true -cp >>>>>> '/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/opt/apache-flume-1.6.0-bin/conf:/opt/apache-flume-1.6.0-bin/lib/*:/lib/*:/home/radu/gits/kite/kite-morphlines/kite-morphlines-all-except-solr/target/lib/*:/opt/solr-5.3.0/dist/*:/opt/solr-5.3.0/dist/solrj-lib/*:/lib/*' >>>>>> -Djava.library.path= org.apache.flume.node.Application -n agent -f >>>>>> conf/flume-conf.properties >>>>>> >>>>>> Here's my Java version, if it matters: >>>>>> java version "1.7.0_79" >>>>>> OpenJDK Runtime Environment (IcedTea 2.5.5) (suse-7.4-x86_64) >>>>>> OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode) >>>>>> >>>>>> From what I can understand out off the Internet (thanks, Internet!) >>>>>> this may be caused by multiple jars having different versions of the same >>>>>> thing. But I can only find it in Morphlines (via grep -RHls in all the >>>>>> classpaths). Maybe I'm bumping into >>>>>> https://issues.apache.org/jira/browse/FLUME-2448 ? I was trying to >>>>>> verify this by compiling Kite 0.12.0 but it failed with: >>>>>> [ERROR] Failed to execute goal >>>>>> org.apache.rat:apache-rat-plugin:0.9:check (default) on project >>>>>> kite-data: >>>>>> Too many files with unapproved license: 2 See RAT report.... >>>>>> >>>>>> So then I compiled only morphlines and that worked, only to bump into >>>>>> the same error. >>>>>> >>>>>> As promised, below is the full exception. For completeness, I'm also >>>>>> including the main config and the morphline config: >>>>>> >>>>>> 17 Sep 2015 12:53:36,232 ERROR [lifecycleSupervisor-1-8] >>>>>> (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) >>>>>> - >>>>>> Unable to start SinkRunner: { >>>>>> policy:org.apache.flume.sink.DefaultSinkProcessor@41a72585 >>>>>> counterGroup:{ name:null counters:{} } } - Exception follows. >>>>>> java.lang.SecurityException: class "javax.servlet.AsyncContext"'s >>>>>> signer information does not match signer information of other classes in >>>>>> the same package >>>>>> at java.lang.ClassLoader.checkCerts(ClassLoader.java:952) >>>>>> at java.lang.ClassLoader.preDefineClass(ClassLoader.java:666) >>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:794) >>>>>> at >>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>>>> at java.lang.ClassLoader.defineClass1(Native Method) >>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800) >>>>>> at >>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>>>>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >>>>>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >>>>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >>>>>> at >>>>>> org.kitesdk.morphline.shaded.com.google.common.reflect.ClassPath$ClassInfo.load(ClassPath.java:224) >>>>>> at >>>>>> org.kitesdk.morphline.api.MorphlineContext.getTopLevelClasses(MorphlineContext.java:176) >>>>>> at >>>>>> org.kitesdk.morphline.api.MorphlineContext.importCommandBuilders(MorphlineContext.java:91) >>>>>> at org.kitesdk.morphline.stdlib.Pipe.<init>(Pipe.java:43) >>>>>> at org.kitesdk.morphline.stdlib.PipeBuilder.build(PipeBuilder.java:40) >>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:126) >>>>>> at org.kitesdk.morphline.base.Compiler.compile(Compiler.java:55) >>>>>> at >>>>>> org.apache.flume.sink.solr.morphline.MorphlineHandlerImpl.configure(MorphlineHandlerImpl.java:101) >>>>>> at >>>>>> org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:97) >>>>>> at >>>>>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) >>>>>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79) >>>>>> at >>>>>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>>>>> at >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>> at >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> >>>>>> >>>>>> > cat conf/flume-conf.properties >>>>>> agent.sources = spoolDir >>>>>> agent.channels = spillableMem >>>>>> agent.sinks = solr >>>>>> >>>>>> agent.sources.spoolDir.type = spooldir >>>>>> agent.sources.spoolDir.spoolDir = /opt/logs >>>>>> agent.sources.spoolDir.batchSize = 1000 >>>>>> agent.sources.spoolDir.channels = spillableMem >>>>>> >>>>>> agent.channels.spillableMem.type = SPILLABLEMEMORY >>>>>> agent.channels.spillableMem.memoryCapacity = 100000 >>>>>> agent.channels.spillableMem.overflowCapacity = 1000000 >>>>>> agent.channels.spillableMem.checkpointDir = /var/run/flume >>>>>> agent.channels.spillableMem.dataDirs = /var/run/flume-data >>>>>> >>>>>> agent.sinks.solr.type = >>>>>> org.apache.flume.sink.solr.morphline.MorphlineSolrSink >>>>>> agent.sinks.solr.channel = spillableMem >>>>>> agent.sinks.solr.morphlineFile = conf/morphline.conf >>>>>> agent.sinks.solr.morphlineId = 1 >>>>>> agent.sinks.solr.batchSize = 10000 >>>>>> >>>>>> > cat conf/morphline.conf >>>>>> SOLR_LOCATOR : { >>>>>> collection : gettingstarted >>>>>> solrUrl : "http://192.168.1.3:8983/solr/"; >>>>>> batchSize : 5000 >>>>>> } >>>>>> >>>>>> morphlines : [ >>>>>> { id : 1 >>>>>> commands : [ >>>>>> { readLine { charset : UTF-8 } } >>>>>> { >>>>>> grok { >>>>>> dictionaryFiles : [conf/grok-patterns] >>>>>> expressions : { >>>>>> message : """%{COMBINEDAPACHELOG}""" >>>>>> } >>>>>> } >>>>>> } >>>>>> # { generateUUID { >>>>>> # field : id >>>>>> # } >>>>>> # } >>>>>> { >>>>>> loadSolr { >>>>>> solrLocator : ${SOLR_LOCATOR} >>>>>> } >>>>>> } >>>>>> ] >>>>>> } >>>>>> ] >>>>>> >>>>>> I also have a grok-patterns file there, but I don't think it's >>>>>> relevant. >>>>>> >>>>>> Any ideas are much appreciated! >>>>>> >>>>>> Thanks and best regards, >>>>>> Radu >>>>>> -- >>>>>> Performance Monitoring * Log Analytics * Search Analytics >>>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>>> >>>>> >>>>> >>>> >>> >> >
