If your creating a new producer for each send instead of reusing a pool of producer connections for each send within your threads that could result in the brokers surpassing your open file limit.
/******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop ********************************************/ On Sep 6, 2014 11:29 AM, "Shlomi Hazan" <shl...@viber.com> wrote: > just VPN'ed into my workstation: > the answer to 5 is [*yes*] > answer to 1,2: > the error's I see on the python client are first timeouts and then message > send failures, using sync send. > > on the controller log: > > ontroller.log.2014-08-26-13:[2014-08-26 13:40:44,317] ERROR > [Controller-1-to-broker-3-send-thread], Controller 1 epoch 3 failed to send > StopReplica request with correlation id 519 to broker > id:3,host:shlomi-kafka-broker-3,port:9092. Reconnecting to broker. > (kafka.controller.RequestSendThread) > controller.log.2014-08-26-13:[2014-08-26 13:40:44,319] ERROR > [Controller-1-to-broker-3-send-thread], Controller 1's connection to broker > id:3,host:shlomi-kafka-broker-3,port:9092 was unsuccessful > (kafka.controller.RequestSendThread) > > on the server log (selected greps): > ... > server.log.2014-08-27-01:[2014-08-27 01:44:23,143] ERROR > [ReplicaFetcherThread-4-2], Error for partition [vpq_android_gcm_h,270] to > broker 2:class kafka.common.NotLeaderForPartitionException > (kafka.server.ReplicaFetcherThread) > ... > server.log.2014-08-27-12:[2014-08-27 12:08:34,638] ERROR Closing socket for > /10.184.150.54 because of error (kafka.network.Processor) > > ... > server.log.2014-08-28-07:[2014-08-28 07:57:35,944] ERROR [KafkaApi-1] > Error > when processing fetch request for partition [vpq_android_gcm_h,184] offset > 8798 from follower with correlation id 0 (kafka.server.KafkaApis) > ... > erver.log.2014-09-03-15:[2014-09-03 15:46:18,220] ERROR > [ReplicaFetcherThread-2-3], Error in fetch Name: FetchRequest; Version: 0; > CorrelationId: 177593; ClientId: ReplicaFetcherThread-2-3; ReplicaId: 1; > MaxWait: 1000 ms; MinBytes: 1 bytes; RequestInfo: [vpq_android_gcm_h,196] > -> PartitionFetchInfo(65283,8388608),[vpq_android_gcm_h,76] -> > PartitionFetchInfo(262787,8388608),[vpq_android_gcm_h,460] -> > PartitionFetchInfo(285709,8388608),[vpq_android_gcm_h,100] -> > PartitionFetchInfo(199405,8388608),[vpq_android_gcm_h,148] -> > PartitionFetchInfo(339032,8388608),[vpq_android_gcm_h,436] -> > PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,124] -> > PartitionFetchInfo(484447,8388608),[vpq_android_gcm_h,484] -> > PartitionFetchInfo(105945,8388608),[vpq_android_gcm_h,340] -> > PartitionFetchInfo(0,8388608),[vpq_android_gcm_h,388] -> > PartitionFetchInfo(9,8388608),[vpq_android_gcm_h,316] -> > PartitionFetchInfo(194766,8388608),[vpq_android_gcm_h,364] -> > PartitionFetchInfo(139897,8388608),[vpq_android_gcm_h,292] -> > PartitionFetchInfo(195408,8388608),[vpq_android_gcm_h,28] -> > PartitionFetchInfo(329961,8388608),[vpq_android_gcm_h,172] -> > PartitionFetchInfo(436959,8388608),[vpq_android_gcm_h,268] -> > PartitionFetchInfo(59827,8388608),[vpq_android_gcm_h,244] -> > PartitionFetchInfo(259731,8388608),[vpq_android_gcm_h,220] -> > PartitionFetchInfo(61669,8388608),[vpq_android_gcm_h,412] -> > PartitionFetchInfo(563609,8388608),[vpq_android_gcm_h,4] -> > PartitionFetchInfo(360336,8388608),[vpq_android_gcm_h,52] -> > PartitionFetchInfo(378533,8388608) (kafka.server.ReplicaFetcherThread) > ... > server.log.2014-09-03-14:[2014-09-03 14:04:18,548] ERROR Error in acceptor > (kafka.network.Acceptor) > ... > > > > On Sat, Sep 6, 2014 at 5:48 PM, Shlomi Hazan <shl...@viber.com> wrote: > > > Hi and sorry for the late response I just got into the weekend and still > > Satdurday here... > > Well, not at my desk but will answer what I can: > > 1. what else on the logs? [*will vpn and check*] > > 2. other broker failure reason? [*"*] > > 3. other broker failure after taking leadership? [*how can I be sure? ask > > another to describe topic?*] > > 4. how do I measure number of connections? [*ls -l /proc/<pid>/fd | grep > > socket | wc -l, also did watch on that*] > > 5. is that number equals the number of {new Producer}? > > 6. how many topics? [*1*] how many partitions [*504*] > > 7. Are u using a partition key? [*yes, I use the python client with* ] > > > > > > > > > > > > > > > > > > > > *class ProducerIdPartitioner(Partitioner): """ Implements a > > partitioner which selects the target partition based on the sending > > producer ID """ def partition(self, key, partitions): size = > > len(partitions) prod_id = int(key) idx = prod_id % > > size return partitions[idx]* > > 8. maybe running into over partitioned topic? [*producer instances is 6 > > machines * 84 procs * 24 threads, but never got to start them all*,*b/c > > of errors*] > > 9. r u running anything else? [*yes, zookeeper*] > > > > additional: > > do you want broker or other config? EC2 instance types? anythying > else?.... > > Thanks, > > Shlomi > > > > > > On Thu, Sep 4, 2014 at 10:02 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > > >> I think it sounds more like another issue than your thinking...the > broker > >> should not be failing like that especially another broker being affected > >> doesn't make sense. > >> > >> What else is in the logs on failure? Is the other broker failing > because > >> of number of files too? Is it happening after it becomes the leader? > >> > >> How are you measuring number of connections? Is this how many producer > >> connections you are opening up yourself (new Producer())? > >> > >> How many topics do you have? How many partitions? > >> > >> Are you using a partition key? Maybe you are running into an over > >> partitioned topic type of scenario where your # of producer instances > >> multiplied by # topics is the problem (where the producer will open up > >> it's > >> own connection over time to brokers for each partition leader). > >> > >> Are you running anything else besides kafka broker on the box? > >> > >> /******************************************* > >> Joe Stein > >> Founder, Principal Consultant > >> Big Data Open Source Security LLC > >> http://www.stealth.ly > >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > >> ********************************************/ > >> > >> > >> On Thu, Sep 4, 2014 at 2:06 PM, Shlomi Hazan <shl...@viber.com> wrote: > >> > >> > I sure did. the reason I am building is trying to patch some. > >> specifically > >> > this : KAFKA-1623. > >> > actually if I felt more confident about scala, I would happily send > you > >> a > >> > patch. > >> > If you don't care screening, just tell me how to prep it for ya and i > >> will. > >> > The bigger problem is running into too many open files when ramping up > >> to > >> > several thousand connection. > >> > going above 4k is hard and around 6K the broker says goodbye. > sometimes > >> > taking a broker friend for a ride.... > >> > This was what led me to the acceptor and it's being slow.. > >> > Have you got an idea why 10K connection should pose a problem. ulimit > >> > checked. not that. ?? > >> > Shlomi > >> > > >> > > >> > On Thu, Sep 4, 2014 at 7:00 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > >> > > >> > > Have you tried using a binary release > >> > > http://kafka.apache.org/downloads.html > >> > > this way you don't have to-do a build? > >> > > > >> > > We build using JDK 6 you should be able to run in 8 (I know for sure > >> 6 & > >> > 7 > >> > > work honestly never tried 8). > >> > > > >> > > I just did a quick test with a broker running on 8 and > >> produced/consumed > >> > a > >> > > few message didn't run into issues... > >> > > > >> > > As for building in JDK 8 => I reproduced your issue and created a > >> ticket > >> > > https://issues.apache.org/jira/browse/KAFKA-1624 patches are > >> welcomed. > >> > > > >> > > /******************************************* > >> > > Joe Stein > >> > > Founder, Principal Consultant > >> > > Big Data Open Source Security LLC > >> > > http://www.stealth.ly > >> > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > >> > > ********************************************/ > >> > > > >> > > > >> > > On Thu, Sep 4, 2014 at 11:22 AM, Shlomi Hazan <shl...@viber.com> > >> wrote: > >> > > > >> > > > ./gradlew -PscalaVersion=2.9.2 clean jar failed with JDK 8. > (error: > >> > error > >> > > > while loading CharSequence, class file > >> > > > > >> '/usr/java/jdk1.8.0_20/jre/lib/rt.jar(java/lang/CharSequence.class)' is > >> > > > broken) > >> > > > I understand there's no escape from installing JDK 7? > >> > > > 10x > >> > > > Shlomi > >> > > > > >> > > > > >> > > > On Thu, Sep 4, 2014 at 6:11 PM, Joe Stein <joe.st...@stealth.ly> > >> > wrote: > >> > > > > >> > > > > When building you need to use the ./gradelw script as Harsha > said. > >> > > > Please > >> > > > > take a look at the README for specific commands and how to run > >> them. > >> > > > > > >> > > > > /******************************************* > >> > > > > Joe Stein > >> > > > > Founder, Principal Consultant > >> > > > > Big Data Open Source Security LLC > >> > > > > http://www.stealth.ly > >> > > > > Twitter: @allthingshadoop < > >> http://www.twitter.com/allthingshadoop> > >> > > > > ********************************************/ > >> > > > > > >> > > > > > >> > > > > On Thu, Sep 4, 2014 at 10:59 AM, Shlomi Hazan <shl...@viber.com > > > >> > > wrote: > >> > > > > > >> > > > > > it failed with JDK 8 so I hoped a newer gradle will maybe do > the > >> > > magic, > >> > > > > and > >> > > > > > stepped into this other problem. > >> > > > > > I assume you will say : > >> > > > > > install JDK 7 and build with our gradle 1.6. > >> > > > > > is it so? > >> > > > > > Shlomi > >> > > > > > > >> > > > > > > >> > > > > > On Thu, Sep 4, 2014 at 5:41 PM, Harsha <ka...@harsha.io> > wrote: > >> > > > > > > >> > > > > > > Did you tried "gradlew" script in kafka source dir. > >> > > > > > > -Harsha > >> > > > > > > > >> > > > > > > On Thu, Sep 4, 2014, at 07:32 AM, Shlomi Hazan wrote: > >> > > > > > > > what gradle version is used to build kafka_2.9.2-0.8.1.1 ? > >> > > > > > > > > >> > > > > > > > tried with v2 and failed with : > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > gradle --stacktrace clean > >> > > > > > > > > >> > > > > > > > FAILURE: Build failed with an exception. > >> > > > > > > > > >> > > > > > > > * Where: > >> > > > > > > > Build file > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > '/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle' > >> > > > > > > > line: 34 > >> > > > > > > > > >> > > > > > > > * What went wrong: > >> > > > > > > > A problem occurred evaluating root project > >> 'kafka-0.8.1.1-src'. > >> > > > > > > > > Could not find method add() for arguments [licenseMain, > >> class > >> > > > > > > > nl.javadude.gradle.plugins.license.License] on task set. > >> > > > > > > > > >> > > > > > > > * Try: > >> > > > > > > > Run with --info or --debug option to get more log output. > >> > > > > > > > > >> > > > > > > > * Exception is: > >> > > > > > > > org.gradle.api.GradleScriptException: A problem occurred > >> > > evaluating > >> > > > > > root > >> > > > > > > > project 'kafka-0.8.1.1-src'. > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:54) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:187) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:39) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:34) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:55) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:470) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.project.AbstractProject.evaluate(AbstractProject.java:79) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.configuration.DefaultBuildConfigurer.configure(DefaultBuildConfigurer.java:31) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:128) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:105) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:85) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:81) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:39) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:29) > >> > > > > > > > at > >> > > > > > > >> org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22) > >> > > > > > > > at org.gradle.launcher.Main.doAction(Main.java:33) > >> > > > > > > > at > >> > > > > org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35) > >> > > > > > > > at > >> org.gradle.launcher.GradleMain.main(GradleMain.java:23) > >> > > > > > > > Caused by: org.gradle.api.internal.MissingMethodException: > >> > Could > >> > > > not > >> > > > > > find > >> > > > > > > > method add() for arguments [licenseMain, class > >> > > > > > > > nl.javadude.gradle.plugins.license.License] on task set. > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.AbstractDynamicObject.methodMissingException(AbstractDynamicObject.java:68) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:56) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:172) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.tasks.DefaultTaskContainer_Decorated.invokeMethod(Unknown > >> > > > > > > > Source) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > nl.javadude.gradle.plugins.license.LicensePlugin$_configureSourceSetRule_closure6_closure18.doCall(LicensePlugin.groovy:117) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:59) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > >> > org.gradle.listener.ActionBroadcast.execute(ActionBroadcast.java:39) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.DefaultDomainObjectCollection.doAdd(DefaultDomainObjectCollection.java:164) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.DefaultDomainObjectCollection.add(DefaultDomainObjectCollection.java:159) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.DefaultNamedDomainObjectCollection.add(DefaultNamedDomainObjectCollection.java:71) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:58) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.AbstractNamedDomainObjectContainer.create(AbstractNamedDomainObjectContainer.java:40) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> > org.gradle.api.plugins.JavaPlugin.configureSourceSets(JavaPlugin.java:80) > >> > > > > > > > at > >> > > org.gradle.api.plugins.JavaPlugin.apply(JavaPlugin.java:68) > >> > > > > > > > at > >> > > org.gradle.api.plugins.JavaPlugin.apply(JavaPlugin.java:46) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.plugins.DefaultPluginContainer.providePlugin(DefaultPluginContainer.java:110) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.plugins.DefaultPluginContainer.addPluginInternal(DefaultPluginContainer.java:69) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.plugins.DefaultPluginContainer.apply(DefaultPluginContainer.java:35) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyPlugin(DefaultObjectConfigurationAction.java:117) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.access$200(DefaultObjectConfigurationAction.java:36) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction$3.run(DefaultObjectConfigurationAction.java:85) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.execute(DefaultObjectConfigurationAction.java:130) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.project.AbstractPluginAware.apply(AbstractPluginAware.java:41) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > build_6i3s72vsr5r4u0gb7m8oo0piuq$_run_closure2.doCall(/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle:34) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.ClosureBackedAction.execute(ClosureBackedAction.java:59) > >> > > > > > > > at > >> > > > > org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:130) > >> > > > > > > > at > >> > > > org.gradle.util.ConfigureUtil.configure(ConfigureUtil.java:91) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.project.AbstractProject.configure(AbstractProject.java:785) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.project.AbstractProject.configure(AbstractProject.java:790) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.project.AbstractProject.subprojects(AbstractProject.java:773) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:225) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:129) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > >> > > > > >> > > org.gradle.groovy.scripts.BasicScript.methodMissing(BasicScript.java:79) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > build_6i3s72vsr5r4u0gb7m8oo0piuq.run(/home/shlomi/0dec0xb/project/vpmb/master/3rdparty/kafka/code/kafka-0.8.1.1-src/build.gradle:33) > >> > > > > > > > at > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:52) > >> > > > > > > > ... 31 more > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > 10x > >> > > > > > > > Shlomi > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
