pan3793 commented on a change in pull request #1302: URL: https://github.com/apache/incubator-kyuubi/pull/1302#discussion_r737372525
########## File path: docs/monitor/logging.md ########## @@ -21,11 +21,224 @@ </div> -# Logging +# Monitoring Kyuubi - Logging System -## Server Logging +Kyuubi uses [Apache Log4j](https://logging.apache.org/log4j/2.x/) for logging. -## Process Logging +In general, there are mainly three components in the Kyuubi architecture that will produce component-oriented logs to help you trace breadcrumbs for SQL workloads against Kyuubi. -## Operation Logging +- Logs of Kyuubi Server +- Logs of Kyuubi Engines +- Operation logs +In addition, a Kyuubi deployment for production usually relies on some other external systems. +For example, both Kyuubi servers and engines will use [Apache Zookeeper](https://zookeeper.apache.org/) for service discovery. +The instructions for external system loggings will not be included in this article. + +## Logs of Kyuubi Server + +Logs of Kyuubi Server show us the activities of the server instance including how start/stop, how does it response client requests, etc. + +### Configuring Server Logging + +#### Basic Configurations + +You can configure it by adding a `log4j.properties` file in the `$KYUUBI_HOME/conf` directory. +One way to start is to make a copy of the existing `log4j.properties.template` located there. + +For example, + +```shell +# cd $KYUUBI_HOME +cp conf/log4j.properties.template conf/log4j.properties +``` + +With or without the above step, by default the server logging will redirect the logs to a file named `kyuubi-${env:USER}-org.apache.kyuubi.server.KyuubiServer-${env:HOSTNAME}.out` under the directory of `$KYUUBI_HOME/logs`. + +For example, you can easily find where the server log goes when staring a Kyuubi server from the console output. + +```shell +$ export SPARK_HOME=/Users/kentyao/Downloads/spark/spark-3.2.0-bin-hadoop3.2 +$ cd ~/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin +$ bin/kyuubi start +``` + +```log +Starting Kyuubi Server from /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin +Warn: Not find kyuubi environment file /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin/conf/kyuubi-env.sh, using default ones... +JAVA_HOME: /Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home +KYUUBI_HOME: /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin +KYUUBI_CONF_DIR: /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin/conf +KYUUBI_LOG_DIR: /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin/logs +KYUUBI_PID_DIR: /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin/pid +KYUUBI_WORK_DIR_ROOT: /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin/work +SPARK_HOME: /Users/kentyao/Downloads/spark/spark-3.2.0-bin-hadoop3.2 +SPARK_CONF_DIR: /Users/kentyao/Downloads/spark/spark-3.2.0-bin-hadoop3.2/conf +HADOOP_CONF_DIR: +Starting org.apache.kyuubi.server.KyuubiServer, logging to /Users/kentyao/svn-kyuubi/v1.3.1-incubating-rc0/apache-kyuubi-1.3.1-incubating-bin/logs/kyuubi-kentyao-org.apache.kyuubi.server.KyuubiServer-hulk.local.out +Welcome to + __ __ __ + /\ \/\ \ /\ \ __ + \ \ \/'/' __ __ __ __ __ __\ \ \____/\_\ + \ \ , < /\ \/\ \/\ \/\ \/\ \/\ \\ \ '__`\/\ \ + \ \ \\`\\ \ \_\ \ \ \_\ \ \ \_\ \\ \ \L\ \ \ \ + \ \_\ \_\/`____ \ \____/\ \____/ \ \_,__/\ \_\ + \/_/\/_/`/___/> \/___/ \/___/ \/___/ \/_/ + /\___/ + \/__/ +``` + +#### KYUUBI_LOG_DIR + +You may also notice that there is an environment variable called `KYUUBI_LOG_DIR` in the above example. + +`KYUUBI_LOG_DIR` determines which folder we want to put our server log files. + +For example, the below command will locate the log files to `/Users/kentyao/tmp`. + +```shell +$ mkdir /Users/kentyao/tmp +$ KYUUBI_LOG_DIR=/Users/kentyao/tmp bin/kyuubi start +``` + +```log +Starting org.apache.kyuubi.server.KyuubiServer, logging to /Users/kentyao/tmp/kyuubi-kentyao-org.apache.kyuubi.server.KyuubiServer-hulk.local.out +``` + +#### KYUUBI_MAX_LOG_FILES + +`KYUUBI_MAX_LOG_FILES` controls how many log files will be remained after a Kyuubi server reboots. + +#### Custom Log4j Settings + +Taking control of `$KYUUBI_HOME/conf/log4j.properties` will also give us the ability of customizing server logging as we want. + +For example, we can disable the console appender and enable the file appender like, + +```properties +log4j.rootCategory=INFO, FA +log4j.appender.FA=org.apache.log4j.FileAppender +log4j.appender.FA.append=false +log4j.appender.FA.file=log/dummy.log +log4j.appender.FA.layout=org.apache.log4j.PatternLayout +log4j.appender.FA.layout.ConversionPattern=%d{HH:mm:ss.SSS} %t %p %c{2}: %m%n +log4j.appender.FA.Threshold=DEBUG +``` + +Then everything goes to `log/dummy.log`. + +## Logs of Spark SQL Engine + +Spark SQL Engine is one type of Kyuubi Engines and also a typical Spark application. +Thus, its logs mainly contain the logs of a Spark Driver. +Meanwhile, it also includes how all the services of an engine start/stop, how does it response the incoming calls from Kyuubi servers, etc. + +In general, when an exception occurs, we are able to find more information and clues in the engine's logs. + +#### Configuring Engine Logging + +Please refer to Apache Spark online documentation https://spark.apache.org/docs/latest/configuration.html#configuring-logging for instructions. Review comment: Can we use markdown hyperlink syntax here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
