[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration
[ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094823#comment-13094823 ] Hudson commented on CASSANDRA-3061: --- Integrated in Cassandra-0.8 #307 (See [https://builds.apache.org/job/Cassandra-0.8/307/]) Optionally skip log4j configuration Patch by tjake; reviewed by Aaron Morton for CASSANDRA-3061 jake : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1163736 Files : * /cassandra/branches/cassandra-0.8/CHANGES.txt * /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AbstractCassandraDaemon.java Optionally skip log4j configuration --- Key: CASSANDRA-3061 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.4 Reporter: Aaron Morton Assignee: T Jake Luciani Priority: Minor Fix For: 0.8.5 Attachments: 0001-3061.patch, 3061_v2.txt from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8 When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. Two examples... http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end. As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file... 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows property namemapred.child.java.opts/name value-Xmx256m -Dcassandra.log4j.configure=false/value description Tune your mapred jvm arguments for best performance. Also see documentation from jvm vendor. /description /property It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone. Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages https://issues.apache.org/jira/browse/HADOOP-7556 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration
[ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13094237#comment-13094237 ] Aaron Morton commented on CASSANDRA-3061: - I tried to run it with the current cassandra-0.8 branch and brisk beta 2 but ran into this problem http://groups.google.com/group/brisk-users/browse_thread/thread/75c9f39d4c1859a9 What I can see looks good to me though. Optionally skip log4j configuration --- Key: CASSANDRA-3061 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.4 Reporter: Aaron Morton Assignee: Aaron Morton Priority: Minor Fix For: 0.8.5 Attachments: 0001-3061.patch, 3061_v2.txt from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8 When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. Two examples... http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end. As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file... 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows property namemapred.child.java.opts/name value-Xmx256m -Dcassandra.log4j.configure=false/value description Tune your mapred jvm arguments for best performance. Also see documentation from jvm vendor. /description /property It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone. Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages https://issues.apache.org/jira/browse/HADOOP-7556 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration
[ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092886#comment-13092886 ] T Jake Luciani commented on CASSANDRA-3061: --- Aaron any update here? this is working fine for me. Optionally skip log4j configuration --- Key: CASSANDRA-3061 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.4 Reporter: Aaron Morton Assignee: Aaron Morton Priority: Minor Fix For: 0.8.5 Attachments: 0001-3061.patch, 3061_v2.txt from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8 When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. Two examples... http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end. As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file... 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows property namemapred.child.java.opts/name value-Xmx256m -Dcassandra.log4j.configure=false/value description Tune your mapred jvm arguments for best performance. Also see documentation from jvm vendor. /description /property It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone. Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages https://issues.apache.org/jira/browse/HADOOP-7556 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration
[ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13088463#comment-13088463 ] T Jake Luciani commented on CASSANDRA-3061: --- Yeah this is a pain. I think it would be easier to check for log4j.defaultInitOverride=true (we set it in bin/cassandra) before we apply our log4j setup. That way in hadoop it would not be loaded by default. Optionally skip log4j configuration --- Key: CASSANDRA-3061 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.4 Reporter: Aaron Morton Assignee: Aaron Morton Priority: Minor Fix For: 0.8.5 Attachments: 0001-3061.patch from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8 When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. Two examples... http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end. As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file... 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows property namemapred.child.java.opts/name value-Xmx256m -Dcassandra.log4j.configure=false/value description Tune your mapred jvm arguments for best performance. Also see documentation from jvm vendor. /description /property It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone. Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages https://issues.apache.org/jira/browse/HADOOP-7556 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3061) Optionally skip log4j configuration
[ https://issues.apache.org/jira/browse/CASSANDRA-3061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13089090#comment-13089090 ] Aaron Morton commented on CASSANDRA-3061: - Thanks, will try it out in the next few days. Optionally skip log4j configuration --- Key: CASSANDRA-3061 URL: https://issues.apache.org/jira/browse/CASSANDRA-3061 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 0.8.4 Reporter: Aaron Morton Assignee: Aaron Morton Priority: Minor Fix For: 0.8.5 Attachments: 0001-3061.patch, 3061_v2.txt from this thread http://groups.google.com/group/brisk-users/browse_thread/thread/3a18f4679673bea8 When brisk accesses cassandra classes inside of a Hadoop Task JVM the AbstractCassandraDaemon uses a log4j PropertyConfigurator to setup cassandra logging. This closes all the existing appenders, including the TaskLogAppender for the hadoop task. They are not opened again because they are not in the config. log4j has Logger Repositories to handle multiple configs in the same process, but there is a bit of suck involved in making a RepositorySelector. Two examples... http://www.mail-archive.com/log4j-dev@jakarta.apache.org/msg02972.html http://docs.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.2/html/Getting_Started_Guide/logging.log4j.reposelect.html Basically all the selector has access to thread local storage, and it looks like normally people get the class loader from the current thread. A thread will inherit it's class loader from the thread that created it, unless otherwise specified. We have code in the same thread the uses hadoop and cassandra classes, so this could be a dead end. As a work around i've added cassandra.log4j.configure JVM param and made the AbstractCassandraServer skip the log4j config if it's false. My job completes and I can see the cassandra code logging an extra message I put in into the Hadoop task log file... 2011-08-19 15:56:06,442 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Metrics system not started: Cannot locate configuration: tried hadoop-metrics2-maptask.properties, hadoop-metrics2.properties 2011-08-19 15:56:06,776 INFO org.apache.cassandra.service.AbstractCassandraDaemon: Logging initialized externally 2011-08-19 15:56:07,332 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 The param has to be passed to the task JVM, so need to modify Haddop mapred-site.xml as follows property namemapred.child.java.opts/name value-Xmx256m -Dcassandra.log4j.configure=false/value description Tune your mapred jvm arguments for best performance. Also see documentation from jvm vendor. /description /property It's not pretty but it works. In my extra log4j logging I can see the second reset() call is gone. Change the to Hadoop TaskLogAppender also stops the NPE but there may also be some lost log messages https://issues.apache.org/jira/browse/HADOOP-7556 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
