We should document how the PluginManager class can be used to generate the Log42Plugins.dat file, and how the packages attribute works. Maybe in Manual > Extending Log4j > Custom Plugins? That section is currently empty...
On Sun, Aug 18, 2013 at 12:57 AM, Ralph Goers <[email protected]>wrote: > Keep in mind, I added the Log4j2Plugins.dat because classpath scanning was > so slow. This would still be done for users who create their own plugins > but don't provide a Log4j2Plugins.dat file, so finding a faster method > could be worthwhile. > > Ralph > > > > On Aug 17, 2013, at 8:44 AM, Nick Williams wrote: > > > On Aug 17, 2013, at 10:28 AM, Remko Popma wrote: > > As part of the build, a plugin database file is generated and included in > the core jar. The file is called Log4j2Plugins.dat and it is located in the > org.apache.logging.log4j.core.config.plugins package. It is in binary > format and contains all classes that define plugins that could be found > during the build. > > > Understood. > > > At load time, the PluginManager class will search for all resources named > org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat in the > classpath. (So there may be multiple jars that each contain a database > file.) This is pretty fast. > > > Then it's unlikely we would see any performance gains of significance. > There's still the problem, though, that the plugin classes in this dat file > are actually _loaded_ eagerly, even if they are never used. My particular > JVM implementation only complained when I started using an exception from a > transitive dependency, but some other JVM (like IBM's or Azul Systems's > JVMs) might complain more aggressively. So there may still be a way we can > improve this without too much effort. > > In addition to this, you can also provide a list of packages that contain > custom plugins in the configuration: > <configuration status="TRACE" packages="com.a.b,com.x.y"> ... > > > It's possible this could also use improvement, once again because loading > classes that are never actually used can cause problems like this. > > N > > I'm not too worried about the performance of this as there is no scanning > or searching outside of these packages in the jars in the classpath. > > > On Sat, Aug 17, 2013 at 11:46 PM, Nick Williams < > [email protected]> wrote: > >> >> On Aug 17, 2013, at 8:20 AM, Gary Gregory wrote: >> >> > Detective Nick is one the case! :) >> >> Thank, Gary! I don't give up. I do NOT like not knowing why something is >> working in an unexpected way. It means I don't know something I should. >> >> > Do those other projects use third party line for this or roll their own? >> >> Tomcat rolls its own, AFAICT. Spring and Hibernate use third-party >> libraries for sure. I'm going to look into what's necessary. My >> understanding is we would get both a startup performance boost /and/ use >> less memory by not loading every class to scan for annotations. >> >> Question: WHAT does Log4j scan to look for plugins? Does it scan every >> class in the JAR (in which case the performance and memory improvements >> would be minor), or does it scan every class on the entire class path (in >> which case the performance and memory improvements would me major)? The >> larger the likely improvements, the more effort we should invest it making >> this happen. >> >> Nick >> >> > On Aug 17, 2013, at 4:17, Nick Williams <[email protected]> >> wrote: >> > >> >> Solved it! >> >> >> >> And you're never gonna believe what I learned tonight...(well, maybe >> you will) >> >> >> >> I solved the error by changing this: >> >> >> >> public final class MongoDBProvider implements >> NoSQLProvider<MongoDBConnection> { >> >> ... >> >> + try { >> >> + if (!database.authenticate(username, >> password.toCharArray())) { >> >> + LOGGER.error("Failed to authenticate against >> MongoDB server. Unknown error."); >> >> + } >> >> + } catch (MongoException e) { >> >> + LOGGER.error("Failed to authenticate against >> MongoDB: " + e.getMessage(), e); >> >> + } catch (IllegalStateException e) { >> >> + LOGGER.error("Factory-supplied MongoDB database >> connection already authenticated with different" + >> >> + "credentials but lost connection."); >> >> + } >> >> ... >> >> } >> >> >> >> To this: >> >> >> >> public final class MongoDBProvider implements >> NoSQLProvider<MongoDBConnection> { >> >> ... >> >> + MongoDBConnection.authenticate(database, username, >> password); >> >> ... >> >> } >> >> >> >> public final class MongoDBConnection implements >> NoSQLConnection<BasicDBObject, MongoDBObject> { >> >> ... >> >> + static void authenticate(final DB database, final String >> username, final String password) { >> >> + try { >> >> + if (!database.authenticate(username, >> password.toCharArray())) { >> >> + LOGGER.error("Failed to authenticate against MongoDB >> server. Unknown error."); >> >> + } >> >> + } catch (final MongoException e) { >> >> + LOGGER.error("Failed to authenticate against MongoDB: " + >> e.getMessage(), e); >> >> + } catch (final IllegalStateException e) { >> >> + LOGGER.error("Factory-supplied MongoDB database >> connection already authenticated with different" + >> >> + "credentials but lost connection."); >> >> + } >> >> + } >> >> ... >> >> } >> >> >> >> Crazy, right!? Here's what I've learned: >> >> >> >> The errors were occurring in tests for the Log4j 1.2 API and the SLF4J >> Bridge. These tests use the core Logger which triggers plugin discovery. In >> order to scan for annotations, plugin discovery loads the MongoDBProvider, >> CouchDBProvider, and JPAAppender classes, among many others, all of which >> have transitive dependencies that are not on the classpath for running the >> unit tests for Log4j 1.2 API and SLF4J. So how did it ever work in the >> first place? >> >> >> >> As you may already know, when Java loads a class it also automatically >> loads any classes it extends or implements, any classes that are the types >> of static members of that class, any static inner classes, and any classes >> used within the static initializer. It doesn't load any other classes that >> class uses in any methods or constructors or as instance members--like >> com.mongodb.DB or javax.persistence.*--until the code that uses them >> actually executes for the first time. Because of this, we can do something >> like load the MongoDBProvider class to scan for @Plugin annotations even >> though the com.mongodb classes it uses are not on the classpath (as long as >> they aren't static members of or extended by the MongoDBProvider, that is). >> >> >> >> However, Java has a special behavior with exceptions. Because we have >> these lovely things called checked exceptions that methods must declare to >> be thrown, exceptions are naturally part of a class's interface. Thus, when >> Java loads a class it must load the exceptions the class's methods might >> throw so that it can complete the interface in memory >> (java.lang.Class.getMethod("someMethod").getExceptionTypes()). Likely for >> performance reasons, it doesn't differentiate between exceptions that are >> actually declared to be thrown and exceptions that are just used (caught). >> ANY dependent classes that are exceptions are loaded when the class loads, >> even if they're just caught exceptions. This is why this all worked until I >> started using an exception from a transitive dependency within a plugin >> class (MongoDBProvider). >> >> >> >> (Incidentally, it's also why more advanced class-scanning projects >> like Spring, Hibernate and Tomcat don't load classes using a ClassLoader >> just to scan for annotations. Instead, they inspect the byte code manually >> to scan for annotations, preventing such class loading errors during >> discovery phases and also saving memory resources since Classes aren't >> usually garbage collected. It might be worthwhile to look into doing >> something similar in Log4j plugin discovery. I don't know how much effort >> would be involved.) >> >> >> >> I haven't confirmed any of this with JLS documentation because no >> amount of Google searching for the combination of "class loading" and >> "exception" brings up anything other than 10,000,000 people asking >> questions about what's wrong with their classpath. I simply can't find that >> needle in a planet full of haystacks. But my thorough experimentation has >> some pretty clear results. This is exactly what's happening. >> >> >> >> Nick >> >> >> >> On Aug 17, 2013, at 12:44 AM, Ralph Goers wrote: >> >> >> >>> I'll reiterate what I wrote. Catch the RuntimeException and then do >> >>> >> >>> if (e.class.getName().equals("com.mongodb.MongoException")) { >> >>> LOGGER.error("..."); >> >>> } else { >> >>> throw e; >> >>> } >> >>> >> >>> This should give you the same behavior. >> >>> >> >>> Ralph >> >>> >> >>> On Aug 16, 2013, at 9:49 PM, Nick Williams wrote: >> >>> >> >>>> That approach concerns me. Catching RuntimeException essentially >> opens it up to thousands of possible exceptions that could be the cause, as >> opposed to looking for that exact cause. I suppose I don't have a choice, >> though. This apparently just isn't going to work. >> >>>> >> >>>> Definitely agreed that there is too much going on for a simple >> Exception class. >> >>>> >> >>>> :-/ >> >>>> >> >>>> Nick >> >>>> >> >>>> On Aug 16, 2013, at 11:45 PM, Ralph Goers wrote: >> >>>> >> >>>>> After following the chain of stuff that gets brought in via >> BSONObject I still recommend the approach in my other email of just >> catching RuntimeException. A bunch of other classes are being referenced, >> one of which is creating a static Logger from java.util.logging. I have no >> idea why that might fail but there is just way too much going on for a >> simple Exception class. >> >>>>> >> >>>>> Ralph >> >>>>> >> >>>>> On Aug 16, 2013, at 9:26 PM, Nick Williams wrote: >> >>>>> >> >>>>>> >> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/DB.java >> >>>>>> >> >>>>>> That also shows an import for org.bson.BSONObject, but the tests >> still run with just DB and no MongoException. org.bson is in the >> org.mongodb:mongo-java-driver JAR file. So, no, that's not the problem. >> There's something else... >> >>>>>> >> >>>>>> Nick >> >>>>>> >> >>>>>> On Aug 16, 2013, at 11:20 PM, Ralph Goers wrote: >> >>>>>> >> >>>>>>> >> https://github.com/mongodb/mongo-java-driver/blob/master/src/main/com/mongodb/MongoException.javashows >> an import for org.bson.BSONObject. The pom.xml for mongo-java-driver >> doesn't contain a transitive dependency for that and mvn dependency:tree on >> core doesn't show it. >> >>>>>>> >> >>>>>>> Ralph >> >>>>>>> >> >>>>>>> >> >>>>>>> On Aug 16, 2013, at 3:48 PM, Nick Williams wrote: >> >>>>>>> >> >>>>>>>> Guys, I'm having a hard time with this simple fix that should >> have taken five minutes. I'm getting test failures due to >> NoClassDefFoundErrors that shouldn't happen. >> >>>>>>>> >> >>>>>>>> Here are the tests in error: >> >>>>>>>> CategoryTest.setupClass:52 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testTraceWithException:415 ? NoClassDefFound >> com/mongodb/MongoExcep... >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testLog:459 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testRB1:295 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testRB2:314 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testRB3:334 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testTrace:388 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testAdditivity1:119 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testAdditivity2:144 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testAdditivity3:183 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testIsTraceEnabled:443 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.testExists:355 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggerTest.tearDown:75 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggingTest.setupClass:44 ? NoClassDefFound >> com/mongodb/MongoException >> >>>>>>>> LoggingTest.cleanupClass:49 NullPointer >> >>>>>>>> >> >>>>>>>> Here's the code I added: >> >>>>>>>> >> >>>>>>>> try { >> >>>>>>>> if (!database.authenticate(username, >> password.toCharArray())) { >> >>>>>>>> LOGGER.error("Failed to authenticate >> against MongoDB server. Unknown error."); >> >>>>>>>> } >> >>>>>>>> } catch (MongoException e) { >> >>>>>>>> LOGGER.error("Failed to authenticate against >> MongoDB: " + e.getMessage(), e); >> >>>>>>>> } catch (IllegalStateException e) { >> >>>>>>>> LOGGER.error("Factory-supplied MongoDB >> database connection already authenticated with different" + >> >>>>>>>> "credentials but lost connection."); >> >>>>>>>> } >> >>>>>>>> >> >>>>>>>> Problem is, "database" is an instance of com.mongodb.DB, which >> is in the same JAR as com.mongodb.MongoException. If I remove this code, >> the tests pass. How is this possible? The DB instance is there with or >> without this new code, which means the JAR is on the classpath, which means >> MongoException should be on the classpath. >> >>>>>>>> >> >>>>>>>> Very confused... >> >>>>>>>> >> >>>>>>>> Nick >> >>>>>>>> >> >>>>>>>> On Aug 16, 2013, at 5:13 PM, Gary Gregory wrote: >> >>>>>>>> >> >>>>>>>>> Thank you for the update Nick! >> >>>>>>>>> :) >> >>>>>>>>> Gary >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Fri, Aug 16, 2013 at 5:39 PM, Nick Williams < >> [email protected]> wrote: >> >>>>>>>>> Answers inline. >> >>>>>>>>> >> >>>>>>>>> On Aug 14, 2013, at 2:10 AM, YuCheng Ting wrote: >> >>>>>>>>> >> >>>>>>>>>> Hi all, >> >>>>>>>>>> >> >>>>>>>>>> I use beta8 log4j2 and wrote log4j2.xml like example in >> document ( >> http://logging.apache.org/log4j/2.x/manual/appenders.html#NoSQLAppender): >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> <appenders> >> >>>>>>>>>> <NoSql name="databaseAppender"> >> >>>>>>>>>> <MongoDb databaseName="applicationDb" >> collectionName="applicationLog" >> >>>>>>>>>> server="mongo.example.org" >> >>>>>>>>>> username="loggingUser" password="abc123" /> >> >>>>>>>>>> </NoSql> >> >>>>>>>>>> </appenders> >> >>>>>>>>> >> >>>>>>>>> Yep. That's correct. >> >>>>>>>>> >> >>>>>>>>>> but I get the two exception: >> >>>>>>>>>> >> >>>>>>>>>> 1, "can't serialize class org.apache.logging.log4j.Level" >> exception in (BasicBSONEncoder.java:270), I read the code and add follow >> code in my project before logging, it gone. >> >>>>>>>>>> >> >>>>>>>>>> >> BSON.addEncodingHook(org.apache.logging.log4j.Level.class, new >> Transformer() { >> >>>>>>>>>> @Override >> >>>>>>>>>> public Object transform(Object o) { >> >>>>>>>>>> return o.toString(); >> >>>>>>>>>> } >> >>>>>>>>>> }); >> >>>>>>>>> >> >>>>>>>>> This bug was reported and fixed a few weeks ago. The fix will >> be in the next version, or you can compile locally. >> https://issues.apache.org/jira/browse/LOG4J2-330 >> >>>>>>>>> >> >>>>>>>>>> 2, “not authorized for insert test.log”, because my MongoDB >> need auth to write, but the the "username" and "password" attributes in >> log4j2.xml is nearly useless, after I read source code, found it NOT auth in >> >>>>>>>>>> >> >>>>>>>>>> >> org.apache.logging.log4j.core.appender.db.nosql.mongo.MongoDBProvider.createNoSQLProvider >> >>>>>>>>>> source code line 181 after check username and password and >> com.mongodb.DB.authenticate never be called. >> >>>>>>>>> >> >>>>>>>>> This is a bug. I'm reporting it and fixing it now. The fix will >> be in the next version, or you can compile locally (after I get the change >> committed, of course). >> >>>>>>>>> >> >>>>>>>>>> so I change log4j2.xml : >> >>>>>>>>>> >> >>>>>>>>>> <NoSql name="mongodb"> >> >>>>>>>>>> <MongoDb collectionName="log" databaseName="test" >> >>>>>>>>>> >> factoryClassName="com.yuchs.test.log4j.MainTest" >> >>>>>>>>>> factoryMethodName="getMongoClient" /> >> >>>>>>>>>> </NoSql> >> >>>>>>>>>> >> >>>>>>>>>> and create MongoClient and call com.mongodb.DB.authenticate >> method in com.yuchs.test.log4j.MainTest.getMongoClient. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> This is my question: >> >>>>>>>>>> >> >>>>>>>>>> 1, Why not add BSON.addEncodingHook code into log4j2 project >> to avoid basic exception ? or another rule of method I don't know ? >> >>>>>>>>>> >> >>>>>>>>>> 2, Why not auth DB in log4j2 project if password and username >> is set in log4j2.xml ? or another rule of method I don't know ? >> >>>>>>>>>> >> >>>>>>>>>> Thanks everyone! >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> -- >> >>>>>>>>> E-Mail: [email protected] | [email protected] >> >>>>>>>>> Java Persistence with Hibernate, Second Edition >> >>>>>>>>> JUnit in Action, Second Edition >> >>>>>>>>> Spring Batch in Action >> >>>>>>>>> Blog: http://garygregory.wordpress.com >> >>>>>>>>> Home: http://garygregory.com/ >> >>>>>>>>> Tweet! http://twitter.com/GaryGregory >> >> >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: [email protected] >> >> For additional commands, e-mail: [email protected] >> >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > >
