[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264009#comment-15264009
]
Matyas Orhidi commented on HIVE-10502:
--
Here is an option that worked for me:
1) Created a log4j config in a folder. (should be beeline-log4j.properties
exactly)
/root/mypreferredclasses/beeline-log4j.properties
--
log4j.rootLogger=TRACE, console
console appender
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=### magic here ### %d{yy/MM/dd
HH:mm:ss} [%t]: %p %c{2}: %m%n
log4j.appender.console.encoding=UTF-8
2) Called the beeline command by adding this folder into the beginning of the
beeline classpath:
export HADOOP_USER_CLASSPATH_FIRST=true; export
HADOOP_CLASSPATH=/root/mypreferredclasses; beeline -u
jdbc:hive2://nosuchhost:1
scan complete in 2ms
Connecting to jdbc:hive2://nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.Utils: Supplied
authorities: nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.Utils: Resolved
authority: nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.HiveConnection: Will try
to open client transport with JDBC Uri: jdbc:hive2://nosuchhost:1
### magic here ### 16/04/29 05:44:42 [main]: DEBUG transport.TSaslTransport:
opening transport org.apache.thrift.transport.TSaslClientTransport@2977cbf7
### magic here ### 16/04/29 05:44:42 [main]: INFO jdbc.HiveConnection: Could
not open client transport with JDBC Uri: jdbc:hive2://nosuchhost:1
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://nosuchhost:1: java.net.UnknownHostException: nosuchhost
(state=08S01,code=0)
Beeline version 1.1.0-cdh5.4.5 by Apache Hive
0: jdbc:hive2://nosuchhost:1 (closed)>
> Cannot specify log4j.properties file location in Beeline
>
>
> Key: HIVE-10502
> URL: https://issues.apache.org/jira/browse/HIVE-10502
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Szehon Ho
>Assignee: Chaoyu Tang
>
> In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
> startup to initialize log4j logging: LogUtils.initHiveLog4j().
> However, seems like this is not the case in Beeline, which also needs log4j
> like as follows:
> {noformat}
> at org.apache.log4j.LogManager.(LogManager.java:127)
> at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
> at
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
> at
> org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
> at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
> at org.apache.hadoop.util.VersionInfo.(VersionInfo.java:37)
> {noformat}
> It would be good to specify it, so it doesn't pick the first one in the
> classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521970#comment-14521970
]
Szehon Ho commented on HIVE-10502:
--
Thanks for the confirmation, Chaoyu. The first part is quite unfortunate,
seems it will affect all the hive scripts and lead to 'hadoop version' printing
out a debug message in a random log4j location (whatever it picks up).
So the second part, I tracked it down to HIVE-8772. [~thejas], the JIRA makes
it so that the log4j properties is hard-coded as 'beeline-log4j.properties' in
conf file, which I don't feel is well-known, at this point would it be valuable
to have it configurable by initializing Log4j in Beeline java code to have the
same user experience as other hive components (--hiveconf hive.log4j.file)?
Cannot specify log4j.properties file location in Beeline
Key: HIVE-10502
URL: https://issues.apache.org/jira/browse/HIVE-10502
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Chaoyu Tang
In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
startup to initialize log4j logging: LogUtils.initHiveLog4j().
However, seems like this is not the case in Beeline, which also needs log4j
like as follows:
{noformat}
at org.apache.log4j.LogManager.clinit(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.VersionInfo.clinit(VersionInfo.java:37)
{noformat}
It would be good to specify it, so it doesn't pick the first one in the
classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522210#comment-14522210
]
Thejas M Nair commented on HIVE-10502:
--
The idea was that almost all users would not want to get that noise from
zookeeper code. WARN level logging in beeline seems appropriate, but INFO level
looks appropriate for HS2, metastore etc. So a different beeline specific log
file I feel is useful.
I am not against making log file name configurable, I am just trying to
understand what you expect users to point it to.
Cannot specify log4j.properties file location in Beeline
Key: HIVE-10502
URL: https://issues.apache.org/jira/browse/HIVE-10502
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Chaoyu Tang
In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
startup to initialize log4j logging: LogUtils.initHiveLog4j().
However, seems like this is not the case in Beeline, which also needs log4j
like as follows:
{noformat}
at org.apache.log4j.LogManager.clinit(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.VersionInfo.clinit(VersionInfo.java:37)
{noformat}
It would be good to specify it, so it doesn't pick the first one in the
classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522253#comment-14522253
]
Szehon Ho commented on HIVE-10502:
--
OK its fine for now, I was just trying to understand the motivation, thanks.
Cannot specify log4j.properties file location in Beeline
Key: HIVE-10502
URL: https://issues.apache.org/jira/browse/HIVE-10502
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Chaoyu Tang
In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
startup to initialize log4j logging: LogUtils.initHiveLog4j().
However, seems like this is not the case in Beeline, which also needs log4j
like as follows:
{noformat}
at org.apache.log4j.LogManager.clinit(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.VersionInfo.clinit(VersionInfo.java:37)
{noformat}
It would be good to specify it, so it doesn't pick the first one in the
classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520250#comment-14520250
]
Chaoyu Tang commented on HIVE-10502:
Beeline seems not use log4j at all and jline2 has its own logging
implementation.
Cannot specify log4j.properties file location in Beeline
Key: HIVE-10502
URL: https://issues.apache.org/jira/browse/HIVE-10502
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Chaoyu Tang
In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
startup to initialize log4j logging: LogUtils.initHiveLog4j().
However, seems like this is not the case in Beeline, which also needs log4j
like as follows:
{noformat}
at org.apache.log4j.LogManager.clinit(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.VersionInfo.clinit(VersionInfo.java:37)
{noformat}
It would be good to specify it, so it doesn't pick the first one in the
classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520386#comment-14520386
]
Szehon Ho commented on HIVE-10502:
--
Hey Chaoyu, did you check the static initialization of the VersionInfo class,
as I posted in the stack? It does attempt to read the log4j.properties file
and picks a random one. But you might be right, it may not log at all via
log4j, can you verify with a few commands whether log4j is ever called?
Cannot specify log4j.properties file location in Beeline
Key: HIVE-10502
URL: https://issues.apache.org/jira/browse/HIVE-10502
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Chaoyu Tang
In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
startup to initialize log4j logging: LogUtils.initHiveLog4j().
However, seems like this is not the case in Beeline, which also needs log4j
like as follows:
{noformat}
at org.apache.log4j.LogManager.clinit(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.VersionInfo.clinit(VersionInfo.java:37)
{noformat}
It would be good to specify it, so it doesn't pick the first one in the
classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10502) Cannot specify log4j.properties file location in Beeline
[
https://issues.apache.org/jira/browse/HIVE-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520723#comment-14520723
]
Chaoyu Tang commented on HIVE-10502:
[~szehon] I see the issue. What you did is to invoke Beeline application from
the shell script 'beeline'. I think in hive shell script, which beeline uses,
calls hadoop cmd to check HADOOP_VERSION if it is not found in env:
HADOOP_VERSION=$($HADOOP version | awk '{if (NR == 1) {print $2;}}');
In script hadoop, it calls java org.apache.hadoop.util.VersionInfo, and try to
locate the log4j.properties file in the path.
Actually in beeline script, log4j.properties is specified as
beeline-log4j.properties and passed to hadoop jar application for Beeline:
export HADOOP_CLIENT_OPTS=$HADOOP_CLIENT_OPTS
-Dlog4j.configuration=beeline-log4j.properties
So you might specify the beeline log4j.properties via
HADOOP_CLIENT_OPTS=-Dlog4j.configuration=anypath/anyfile.properties ?
Cannot specify log4j.properties file location in Beeline
Key: HIVE-10502
URL: https://issues.apache.org/jira/browse/HIVE-10502
Project: Hive
Issue Type: Bug
Components: Beeline
Affects Versions: 1.1.0
Reporter: Szehon Ho
Assignee: Chaoyu Tang
In HiveCLI, HiveServer2, HMS, etc, the following is called early in the
startup to initialize log4j logging: LogUtils.initHiveLog4j().
However, seems like this is not the case in Beeline, which also needs log4j
like as follows:
{noformat}
at org.apache.log4j.LogManager.clinit(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:270)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:156)
at
org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.hadoop.util.VersionInfo.clinit(VersionInfo.java:37)
{noformat}
It would be good to specify it, so it doesn't pick the first one in the
classpath.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
