Logback predefines a variable HOSTNAME if you want to use it anywhere in
your logback.xml
Here's what I have in my logback -
<appender name="FILE" class=
"ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/drill/drillbit_${HOSTNAME}.log</file>
<rollingPolicy class=
"ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>
/var/log/drill/logs/archive/drillbit_${HOSTNAME}.log.%i</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>10</maxIndex>
</rollingPolicy>
On Mon, Oct 12, 2015 at 6:02 PM, Jacques Nadeau <[email protected]> wrote:
> I don't have a specific answer but the logging is defined in this file:
>
>
> https://github.com/apache/drill/blob/master/distribution/src/resources/logback.xml
>
> You can see that there are two loggers, one named QUERY and one named FILE.
> Log query path and log path come from the two environment variables you've
> identified.
>
> I'm guessing that the issue you're having has something to do with a
> failure to propagate these values.
>
> With regards to hostname in logfile, I believe logback exposes a parameter
> that can be included in the logback.xml, as we've used this before. Maybe
> Kunal or Steven (cc'd) remembers more on this.
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Mon, Oct 12, 2015 at 6:13 AM, John Omernik <[email protected]> wrote:
>
> > A follow-up, since I am using "runbit" The line below exists. It sources
> > drill-config.sh which then sources drill-env.sh. In neither of these
> files
> > do I see the variables DRILLBIT_LOG_PATH and DRILLBIT_QUERY_LOG_PATH so
> > that may be the "file not found" error I am getting? If I set those via
> > export in drill-env.sh, to be directories, then I get "File not found
> > exceptions" but this time it shows the path that I am setting and claims
> > the error is because the "Path" is a directory. I guess, am I seeing
> some
> > sort of bug due to what is expected by runbit vs. drillbit.sh. Looking at
> > drillbit.sh, it looks like hit has more settings such as DRILL_OUTFILE,
> > DRILL_LOGFILE, and DRILL_QUERY_FILE none of which exist in runbit.
> Should
> > we be having drillbit.sh and runbit be more consistent?
> >
> >
> >
> > exec $JAVA -Dlog.path=$DRILLBIT_LOG_PATH -Dlog.query.path=
> > $DRILLBIT_QUERY_LOG_PATH $DRILL_ALL_JAVA_OPTS -cp $CP
> > org.apache.drill.exec.server.Drillbit
> >
> > On Thu, Oct 8, 2015 at 12:09 PM, John Omernik <[email protected]> wrote:
> >
> > > Hello all -
> > >
> > > I am playing with Drill (I have 1.2 running right now) and I am trying
> to
> > > figure out a way to do some logging in a logical way. I know I am
> > running
> > > outside the norm, in that I am running my drill bits using marathon on
> > > mesos, but for a moment (unless it's THAT that is breaking my logging
> :)
> > > ignore that fact :)
> > >
> > > I am using MapR, which is nice because it lets me NFS mount my
> > distributed
> > > file system (MapRFS) Thus, when I run a drillbit, I package up the
> > > drill-1.2.0 directory (I am using the MapR packaged version) and then
> in
> > > Marathon I create a app profile like this
> > >
> > > {
> > > "cmd": "./drill-1.2.0/bin/runbit --config
> > > /mapr/zetapoc/mesos/prod/drill/conf.std",
> > > "cpus": 12.0,
> > > "mem": 22528,
> > > "id": "zetadrill",
> > > "instances": 5,
> > > "uris":["hdfs:///mesos/prod/drill/drill-1.2.0.tgz"],
> > > "constraints": [["hostname", "UNIQUE"]]
> > > }
> > >
> > > Pretty basic, for non Mesos folks, it takes the URI (the tgz) downloads
> > it
> > > to the Mesos Sandbox, untars it there, and runs 'runbit' using cgroups.
> > > Thus the memory and cpu specifications. I am passing it a full path to
> > the
> > > config location which is just the /conf copied out of the drill-1.2.0
> > > directory and modified. Thus, all bits are pulling from the same conf.
> (I
> > > could package the conf in the tgz, this makes its easy to change a
> config
> > > and not have to repackage the tgz)
> > >
> > > Also, as you can see I am using the constraint to only have one per
> node,
> > > this is handy from a port conflict point of view.
> > >
> > > Ok, that out of the way, when I run it, in my drill-env.sh, I am
> setting:
> > > export DRILL_LOG_DIR="./drill-1.2.0/log"
> > >
> > > What I "think" that means is it should log inside the sandbox. And sure
> > > enough, when I look in the sandbox in mesos, in the log directory,
> there
> > is
> > > a profiles folder and the sys.drill (query logs... I think?)
> > >
> > > Great.
> > >
> > > Now, in the stdout on each node that started drill, I do have the error
> > > below... and additionally, I don't see the drillbit.log and
> drillbit.out
> > > files. (Is that because of the errors below?) Wouldn't these be in
> > > ./drill-1.2.0/log as well?
> > >
> > > Side note: If I am doing a CTAS to Parquet, one of my nodes starts
> > > spitting out INFO logs to the console like crazy during the write, not
> > sure
> > > what that is all about...
> > >
> > > Ok, back to logging. I am trying to organize this, ideally, I want to
> > > understand logging deeply to meet a number of requirements
> > >
> > > 1. I'd like all profiles/sql audit logs logged to one place. Ideally,
> > > this would make it so if I click profiles on any node, I'd see the
> > > historical queries for all nodes, not just the one I am connected to.
> Is
> > > this possible? Right now, I am only seeing the historical queries if I
> am
> > > connected to a node that acted as foreman for that query
> > >
> > > 2. I'd like to keep all my logs in MapRFS with NFS, thus I'd like all
> > > nodes, for logfiles that have the same names (like drillbit.log and
> > > drillbit.out) to write to a file in the same networked location, but
> have
> > > the hostname in the logfile name. This would allow me to log to the
> same
> > > place on all nodes, but I'd need to know how to do the hostname thing
> (if
> > > that is possible)
> > >
> > > 3. Are there any gotchas to what I am trying to do? What about having
> > > more control over pruning? i.e. Can I keep the sql audit logs in a
> > > directory, and those will show up in the console, but have an job or
> > > setting (inside or outside drill) to move them to a "final" resting
> place
> > > for longterm auditing?
> > >
> > > I know some of these things may be obvious to users of drill, I have
> been
> > > having a hard time digging through the docs (for example, I am not sure
> > > where to find a good explanation of ENV vars that may help me here),
> any
> > > advice would be welcome!
> > >
> > > John
> > >
> > >
> > >
> > > Error on in STDOUT on sandbox.. is this hurting me?
> > >
> > >
> > > 16:35:22,996 |-INFO in
> > > ch.qos.logback.core.rolling.RollingFileAppender[QUERY] - Active log
> file
> > > name:
> > >
> > > 16:35:22,996 |-INFO in
> > > ch.qos.logback.core.rolling.RollingFileAppender[QUERY] - File property
> is
> > > set to []
> > >
> > > 16:35:22,998 |-ERROR in
> > > ch.qos.logback.core.rolling.RollingFileAppender[QUERY] -
> openFile(,true)
> > > call failed. java.io.FileNotFoundException: (No such file or
> directory)
> > >
> > > at java.io.FileNotFoundException: (No such file or directory)
> > >
> > > at at java.io.FileOutputStream.open(Native Method)
> > >
> > > at at
> > > java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:28)
> > >
> > > at at
> > > ch.qos.logback.core.FileAppender.openFile(FileAppender.java:149)
> > >
> > > at at
> > > ch.qos.logback.core.FileAppender.start(FileAppender.java:108)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:86)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:96)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:317)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:196)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:182)
> > >
> > > at at
> > > ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
> > >
> > > at at
> > >
> >
> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
> > >
> > > at at
> > >
> >
> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:148)
> > >
> > > at at
> > > org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
> > >
> > > at at
> > > org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
> > >
> > > at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:129)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:108)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:302)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:276)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288)
> > >
> > > at at
> > > org.apache.drill.exec.server.Drillbit.<clinit>(Drillbit.java:50)
> > >
> > > 16:35:22,998 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
> -
> > > About to instantiate appender of type
> > > [ch.qos.logback.core.rolling.RollingFileAppender]
> > >
> > > 16:35:22,998 |-INFO in ch.qos.logback.core.joran.action.AppenderAction
> -
> > > Naming appender as [FILE]
> > >
> > > 16:35:23,000 |-INFO in
> > > ch.qos.logback.core.rolling.FixedWindowRollingPolicy@6982bec3 - No
> > > compression will be used
> > >
> > > 16:35:23,001 |-INFO in
> > > ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming
> > default
> > > type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for
> [encoder]
> > > property
> > >
> > > 16:35:23,002 |-INFO in
> > > ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file
> > > name:
> > >
> > > 16:35:23,002 |-INFO in
> > > ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property
> is
> > > set to []
> > >
> > > 16:35:23,002 |-ERROR in
> > > ch.qos.logback.core.rolling.RollingFileAppender[FILE] - openFile(,true)
> > > call failed. java.io.FileNotFoundException: (No such file or
> directory)
> > >
> > > at java.io.FileNotFoundException: (No such file or directory)
> > >
> > > at at java.io.FileOutputStream.open(Native Method)
> > >
> > > at at
> > > java.io.FileOutputStream.<init>(FileOutputStream.java:221)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:28)
> > >
> > > at at
> > > ch.qos.logback.core.FileAppender.openFile(FileAppender.java:149)
> > >
> > > at at
> > > ch.qos.logback.core.FileAppender.start(FileAppender.java:108)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:86)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:96)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:317)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:196)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:182)
> > >
> > > at at
> > > ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
> > >
> > > at at
> > >
> >
> ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
> > >
> > > at at
> > >
> >
> ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
> > >
> > > at at
> > >
> >
> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:148)
> > >
> > > at at
> > > org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
> > >
> > > at at
> > > org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
> > >
> > > at at org.slf4j.LoggerFactory.bind(LoggerFactory.java:129)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:108)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:302)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:276)
> > >
> > > at at
> > > org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:288)
> > >
> > > at at
> > > org.apache.drill.exec.server.Drillbit.<clinit>(Drillbit.java:50)
> > >
> >
>
