On Aug 17, 2013, at 10:28 AM, Remko Popma wrote: > As part of the build, a plugin database file is generated and included in the > core jar. The file is called Log4j2Plugins.dat and it is located in the > org.apache.logging.log4j.core.config.plugins package. It is in binary format > and contains all classes that define plugins that could be found during the > build.
Understood. > > At load time, the PluginManager class will search for all resources named > org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat in the > classpath. (So there may be multiple jars that each contain a database file.) > This is pretty fast. Then it's unlikely we would see any performance gains of significance. There's still the problem, though, that the plugin classes in this dat file are actually _loaded_ eagerly, even if they are never used. My particular JVM implementation only complained when I started using an exception from a transitive dependency, but some other JVM (like IBM's or Azul Systems's JVMs) might complain more aggressively. So there may still be a way we can improve this without too much effort. > In addition to this, you can also provide a list of packages that contain > custom plugins in the configuration: > <configuration status="TRACE" packages="com.a.b,com.x.y"> ... It's possible this could also use improvement, once again because loading classes that are never actually used can cause problems like this. N > I'm not too worried about the performance of this as there is no scanning or > searching outside of these packages in the jars in the classpath. > > > On Sat, Aug 17, 2013 at 11:46 PM, Nick Williams > <[email protected]> wrote: > > On Aug 17, 2013, at 8:20 AM, Gary Gregory wrote: > > > Detective Nick is one the case! :) > > Thank, Gary! I don't give up. I do NOT like not knowing why something is > working in an unexpected way. It means I don't know something I should. > > > Do those other projects use third party line for this or roll their own? > > Tomcat rolls its own, AFAICT. Spring and Hibernate use third-party libraries > for sure. I'm going to look into what's necessary. My understanding is we > would get both a startup performance boost /and/ use less memory by not > loading every class to scan for annotations. > > Question: WHAT does Log4j scan to look for plugins? Does it scan every class > in the JAR (in which case the performance and memory improvements would be > minor), or does it scan every class on the entire class path (in which case > the performance and memory improvements would me major)? The larger the > likely improvements, the more effort we should invest it making this happen. > > Nick > > > On Aug 17, 2013, at 4:17, Nick Williams <[email protected]> > > wrote: > > > >> Solved it! > >> > >> And you're never gonna believe what I learned tonight...(well, maybe you > >> will) > >> > >> I solved the error by changing this: > >> > >> public final class MongoDBProvider implements > >> NoSQLProvider<MongoDBConnection> { > >> ... > >> + try { > >> + if (!database.authenticate(username, > >> password.toCharArray())) { > >> + LOGGER.error("Failed to authenticate against > >> MongoDB server. Unknown error."); > >> + } > >> + } catch (MongoException e) { > >> + LOGGER.error("Failed to authenticate against MongoDB: > >> " + e.getMessage(), e); > >> + } catch (IllegalStateException e) { > >> + LOGGER.error("Factory-supplied MongoDB database > >> connection already authenticated with different" + > >> + "credentials but lost connection."); > >> + } > >> ... > >> } > >> > >> To this: > >> > >> public final class MongoDBProvider implements > >> NoSQLProvider<MongoDBConnection> { > >> ... > >> + MongoDBConnection.authenticate(database, username, > >> password); > >> ... > >> } > >> > >> public final class MongoDBConnection implements > >> NoSQLConnection<BasicDBObject, MongoDBObject> { > >> ... > >> + static void authenticate(final DB database, final String username, > >> final String password) { > >> + try { > >> + if (!database.authenticate(username, password.toCharArray())) > >> { > >> + LOGGER.error("Failed to authenticate against MongoDB > >> server. Unknown error."); > >> + } > >> + } catch (final MongoException e) { > >> + LOGGER.error("Failed to authenticate against MongoDB: " + > >> e.getMessage(), e); > >> + } catch (final IllegalStateException e) { > >> + LOGGER.error("Factory-supplied MongoDB database connection > >> already authenticated with different" + > >> + "credentials but lost connection."); > >> + } > >> + } > >> ... > >> } > >> > >> Crazy, right!? Here's what I've learned: > >> > >> The errors were occurring in tests for the Log4j 1.2 API and the SLF4J > >> Bridge. These tests use the core Logger which triggers plugin discovery. > >> In order to scan for annotations, plugin discovery loads the > >> MongoDBProvider, CouchDBProvider, and JPAAppender classes, among many > >> others, all of which have transitive dependencies that are not on the > >> classpath for running the unit tests for Log4j 1.2 API and SLF4J. So how > >> did it ever work in the first place? > >> > >> As you may already know, when Java loads a class it also automatically > >> loads any classes it extends or implements, any classes that are the types > >> of static members of that class, any static inner classes, and any classes > >> used within the static initializer. It doesn't load any other classes that > >> class uses in any methods or constructors or as instance members--like > >> com.mongodb.DB or javax.persistence.*--until the code that uses them > >> actually executes for the first time. Because of this, we can do something > >> like load the MongoDBProvider class to scan for @Plugin annotations even > >> though the com.mongodb classes it uses are not on the classpath (as long > >> as they aren't static members of or extended by the MongoDBProvider, that > >> is). > >> > >> However, Java has a special behavior with exceptions. Because we have > >> these lovely things called checked exceptions that methods must declare to > >> be thrown, exceptions are naturally part of a class's interface. Thus, > >> when Java loads a class it must load the exceptions the class's methods > >> might throw so that it can complete the interface in memory > >> (java.lang.Class.getMethod("someMethod").getExceptionTypes()). Likely for > >> performance reasons, it doesn't differentiate between exceptions that are > >> actually declared to be thrown and exceptions that are just used (caught). > >> ANY dependent classes that are exceptions are loaded when the class loads, > >> even if they're just caught exceptions. This is why this all worked until > >> I started using an exception from a transitive dependency within a plugin > >> class (MongoDBProvider). > >> > >> (Incidentally, it's also why more advanced class-scanning projects like > >> Spring, Hibernate and Tomcat don't load classes using a ClassLoader just > >> to scan for annotations. Instead, they inspect the byte code manually to > >> scan for annotations, preventing such class loading errors during > >> discovery phases and also saving memory resources since Classes aren't > >> usually garbage collected. It might be worthwhile to look into doing > >> something similar in Log4j plugin discovery. I don't know how much effort > >> would be involved.) > >> > >> I haven't confirmed any of this with JLS documentation because no amount > >> of Google searching for the combination of "class loading" and "exception" > >> brings up anything other than 10,000,000 people asking questions about > >> what's wrong with their classpath. I simply can't find that needle in a > >> planet full of haystacks. But my thorough experimentation has some pretty > >> clear results. This is exactly what's happening. > >> > >> Nick > >> > >> On Aug 17, 2013, at 12:44 AM, Ralph Goers wrote: > >> > >>> I'll reiterate what I wrote. Catch the RuntimeException and then do > >>> > >>> if (e.class.getName().equals("com.mongodb.MongoException")) { > >>> LOGGER.error("..."); > >>> } else { > >>> throw e; > >>> } > >>> > >>> This should give you the same behavior. > >>> > >>> Ralph > >>> > >>> On Aug 16, 2013, at 9:49 PM, Nick Williams wrote: > >>> > >>>> That approach concerns me. Catching RuntimeException essentially opens > >>>> it up to thousands of possible exceptions that could be the cause, as > >>>> opposed to looking for that exact cause. I suppose I don't have a > >>>> choice, though. This apparently just isn't going to work. > >>>> > >>>> Definitely agreed that there is too much going on for a simple Exception > >>>> class. > >>>> > >>>> :-/ > >>>> > >>>> Nick > >>>> > >>>> On Aug 16, 2013, at 11:45 PM, Ralph Goers wrote: > >>>> > >>>>> After following the chain of stuff that gets brought in via BSONObject > >>>>> I still recommend the approach in my other email of just catching > >>>>> RuntimeException. A bunch of other classes are being referenced, one > >>>>> of which is creating a static Logger from java.util.logging. I have no > >>>>> idea why that might fail but there is just way too much going on for a > >>>>> simple Exception class. > >>>>> > >>>>> Ralph > >>>>> > >>>>> On Aug 16, 2013, at 9:26 PM, Nick Williams wrote: > >>>>> > >>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/DB.java > >>>>>> > >>>>>> That also shows an import for org.bson.BSONObject, but the tests still > >>>>>> run with just DB and no MongoException. org.bson is in the > >>>>>> org.mongodb:mongo-java-driver JAR file. So, no, that's not the > >>>>>> problem. There's something else... > >>>>>> > >>>>>> Nick > >>>>>> > >>>>>> On Aug 16, 2013, at 11:20 PM, Ralph Goers wrote: > >>>>>> > >>>>>>> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/MongoException.java > >>>>>>> shows an import for org.bson.BSONObject. The pom.xml for > >>>>>>> mongo-java-driver doesn't contain a transitive dependency for that > >>>>>>> and mvn dependency:tree on core doesn't show it. > >>>>>>> > >>>>>>> Ralph > >>>>>>> > >>>>>>> > >>>>>>> On Aug 16, 2013, at 3:48 PM, Nick Williams wrote: > >>>>>>> > >>>>>>>> Guys, I'm having a hard time with this simple fix that should have > >>>>>>>> taken five minutes. I'm getting test failures due to > >>>>>>>> NoClassDefFoundErrors that shouldn't happen. > >>>>>>>> > >>>>>>>> Here are the tests in error: > >>>>>>>> CategoryTest.setupClass:52 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoException > >>>>>>>> LoggerTest.testTraceWithException:415 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoExcep... > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testLog:459 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testRB1:295 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testRB2:314 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testRB3:334 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testTrace:388 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testAdditivity1:119 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testAdditivity2:144 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testAdditivity3:183 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testIsTraceEnabled:443 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggerTest.testExists:355 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoException > >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound com/mongodb/MongoException > >>>>>>>> LoggingTest.setupClass:44 ? NoClassDefFound > >>>>>>>> com/mongodb/MongoException > >>>>>>>> LoggingTest.cleanupClass:49 NullPointer > >>>>>>>> > >>>>>>>> Here's the code I added: > >>>>>>>> > >>>>>>>> try { > >>>>>>>> if (!database.authenticate(username, > >>>>>>>> password.toCharArray())) { > >>>>>>>> LOGGER.error("Failed to authenticate against > >>>>>>>> MongoDB server. Unknown error."); > >>>>>>>> } > >>>>>>>> } catch (MongoException e) { > >>>>>>>> LOGGER.error("Failed to authenticate against > >>>>>>>> MongoDB: " + e.getMessage(), e); > >>>>>>>> } catch (IllegalStateException e) { > >>>>>>>> LOGGER.error("Factory-supplied MongoDB database > >>>>>>>> connection already authenticated with different" + > >>>>>>>> "credentials but lost connection."); > >>>>>>>> } > >>>>>>>> > >>>>>>>> Problem is, "database" is an instance of com.mongodb.DB, which is in > >>>>>>>> the same JAR as com.mongodb.MongoException. If I remove this code, > >>>>>>>> the tests pass. How is this possible? The DB instance is there with > >>>>>>>> or without this new code, which means the JAR is on the classpath, > >>>>>>>> which means MongoException should be on the classpath. > >>>>>>>> > >>>>>>>> Very confused... > >>>>>>>> > >>>>>>>> Nick > >>>>>>>> > >>>>>>>> On Aug 16, 2013, at 5:13 PM, Gary Gregory wrote: > >>>>>>>> > >>>>>>>>> Thank you for the update Nick! > >>>>>>>>> :) > >>>>>>>>> Gary > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Fri, Aug 16, 2013 at 5:39 PM, Nick Williams > >>>>>>>>> <[email protected]> wrote: > >>>>>>>>> Answers inline. > >>>>>>>>> > >>>>>>>>> On Aug 14, 2013, at 2:10 AM, YuCheng Ting wrote: > >>>>>>>>> > >>>>>>>>>> Hi all, > >>>>>>>>>> > >>>>>>>>>> I use beta8 log4j2 and wrote log4j2.xml like example in document > >>>>>>>>>> (http://logging.apache.org/log4j/2.x/manual/appenders.html#NoSQLAppender > >>>>>>>>>> ): > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> <appenders> > >>>>>>>>>> <NoSql name="databaseAppender"> > >>>>>>>>>> <MongoDb databaseName="applicationDb" > >>>>>>>>>> collectionName="applicationLog" > >>>>>>>>>> server="mongo.example.org" > >>>>>>>>>> username="loggingUser" password="abc123" /> > >>>>>>>>>> </NoSql> > >>>>>>>>>> </appenders> > >>>>>>>>> > >>>>>>>>> Yep. That's correct. > >>>>>>>>> > >>>>>>>>>> but I get the two exception: > >>>>>>>>>> > >>>>>>>>>> 1, "can't serialize class org.apache.logging.log4j.Level" > >>>>>>>>>> exception in (BasicBSONEncoder.java:270), I read the code and add > >>>>>>>>>> follow code in my project before logging, it gone. > >>>>>>>>>> > >>>>>>>>>> BSON.addEncodingHook(org.apache.logging.log4j.Level.class, > >>>>>>>>>> new Transformer() { > >>>>>>>>>> @Override > >>>>>>>>>> public Object transform(Object o) { > >>>>>>>>>> return o.toString(); > >>>>>>>>>> } > >>>>>>>>>> }); > >>>>>>>>> > >>>>>>>>> This bug was reported and fixed a few weeks ago. The fix will be in > >>>>>>>>> the next version, or you can compile locally. > >>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-330 > >>>>>>>>> > >>>>>>>>>> 2, “not authorized for insert test.log”, because my MongoDB need > >>>>>>>>>> auth to write, but the the "username" and "password" attributes in > >>>>>>>>>> log4j2.xml is nearly useless, after I read source code, found it > >>>>>>>>>> NOT auth in > >>>>>>>>>> > >>>>>>>>>> org.apache.logging.log4j.core.appender.db.nosql.mongo.MongoDBProvider.createNoSQLProvider > >>>>>>>>>> source code line 181 after check username and password and > >>>>>>>>>> com.mongodb.DB.authenticate never be called. > >>>>>>>>> > >>>>>>>>> This is a bug. I'm reporting it and fixing it now. The fix will be > >>>>>>>>> in the next version, or you can compile locally (after I get the > >>>>>>>>> change committed, of course). > >>>>>>>>> > >>>>>>>>>> so I change log4j2.xml : > >>>>>>>>>> > >>>>>>>>>> <NoSql name="mongodb"> > >>>>>>>>>> <MongoDb collectionName="log" databaseName="test" > >>>>>>>>>> > >>>>>>>>>> factoryClassName="com.yuchs.test.log4j.MainTest" > >>>>>>>>>> factoryMethodName="getMongoClient" /> > >>>>>>>>>> </NoSql> > >>>>>>>>>> > >>>>>>>>>> and create MongoClient and call com.mongodb.DB.authenticate method > >>>>>>>>>> in com.yuchs.test.log4j.MainTest.getMongoClient. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> This is my question: > >>>>>>>>>> > >>>>>>>>>> 1, Why not add BSON.addEncodingHook code into log4j2 project to > >>>>>>>>>> avoid basic exception ? or another rule of method I don't know ? > >>>>>>>>>> > >>>>>>>>>> 2, Why not auth DB in log4j2 project if password and username is > >>>>>>>>>> set in log4j2.xml ? or another rule of method I don't know ? > >>>>>>>>>> > >>>>>>>>>> Thanks everyone! > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> E-Mail: [email protected] | [email protected] > >>>>>>>>> Java Persistence with Hibernate, Second Edition > >>>>>>>>> JUnit in Action, Second Edition > >>>>>>>>> Spring Batch in Action > >>>>>>>>> Blog: http://garygregory.wordpress.com > >>>>>>>>> Home: http://garygregory.com/ > >>>>>>>>> Tweet! http://twitter.com/GaryGregory > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
