alexeykudinkin commented on code in PR #6170:
URL: https://github.com/apache/hudi/pull/6170#discussion_r940795817
##########
hudi-client/hudi-client-common/pom.xml:
##########
@@ -193,6 +193,12 @@
</dependency>
<!-- Test -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-tests-common</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
Review Comment:
We should extract these deps as well to "hudi-tests-common". while we're at
it
##########
hudi-cli/pom.xml:
##########
@@ -212,23 +212,14 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
</dependency>
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-core</artifactId>
- <scope>test</scope>
- </dependency>
-
<dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-api</artifactId>
- <scope>test</scope>
+ <groupId>org.slf4j</groupId>
+ <artifactId>jul-to-slf4j</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
- <scope>test</scope>
Review Comment:
We should not package any logging dependencies (should be provided along w/
query engine)
##########
.github/workflows/bot.yml:
##########
@@ -9,6 +9,8 @@ on:
branches:
- master
- 'release-*'
+env:
+ MVN_ARGS: -ntp -B -V -Pwarn-log
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.shade=warn
-Dorg.slf4j.simpleLogger.log.org.apache.maven.plugins.dependency=warn
Review Comment:
Can we add these overrides to the config instead of needing to specify them
here?
##########
hudi-examples/hudi-examples-spark/pom.xml:
##########
@@ -230,6 +230,27 @@
</exclusion>
</exclusions>
</dependency>
+ <!-- Logging dependencies -->
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
Review Comment:
Why do we need these here?
These should be coming from Spark
##########
hudi-integ-test/pom.xml:
##########
@@ -525,24 +519,9 @@
</executions>
</plugin>
<plugin>
- <groupId>org.scalatest</groupId>
Review Comment:
Why deleting this config?
##########
hudi-spark-datasource/hudi-spark/pom.xml:
##########
@@ -267,6 +252,14 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
</dependency>
+ <dependency>
Review Comment:
We should
1. Designate all these deps as "provided" in the top-level pom
2. Add them to "hudi-common" (therefore avoiding need to replicate them
across the board)
3. Override them as compile dependency in a few places (where they are not
brought in by the query-engine)
##########
docker/demo/config/log4j2.properties:
##########
@@ -0,0 +1,60 @@
+###
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###
+status = warn
+name = HudiConsoleLog
+
+# Set everything to be logged to the console
+appender.console.type = Console
+appender.console.name = CONSOLE
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+
+# Root logger level
+rootLogger.level = warn
+# Root logger referring to console appender
+rootLogger.appenderRef.stdout.ref = CONSOLE
Review Comment:
Do we need to specify stderr or would it fallback to stdout when not set?
##########
hudi-client/hudi-java-client/src/main/resources/log4j2.properties:
##########
@@ -15,9 +15,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
###
-log4j.rootLogger=INFO, A1
-# A1 is set to be a ConsoleAppender.
-log4j.appender.A1=org.apache.log4j.ConsoleAppender
-# A1 uses PatternLayout.
-log4j.appender.A1.layout=org.apache.log4j.PatternLayout
-log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
+status = warn
+name = HudiConsoleLog
Review Comment:
And here
##########
hudi-client/hudi-flink-client/src/main/resources/log4j2.properties:
##########
@@ -15,11 +15,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
###
-log4j.rootLogger=WARN, A1
-log4j.category.org.apache=INFO
-log4j.category.org.apache.parquet.hadoop=WARN
-# A1 is set to be a ConsoleAppender.
-log4j.appender.A1=org.apache.log4j.ConsoleAppender
-# A1 uses PatternLayout.
-log4j.appender.A1.layout=org.apache.log4j.PatternLayout
-log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
+status = warn
+name = HudiConsoleLog
Review Comment:
Same comment as above for Spark client
##########
docker/demo/config/log4j2.properties:
##########
@@ -0,0 +1,60 @@
+###
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+###
+status = warn
+name = HudiConsoleLog
+
+# Set everything to be logged to the console
+appender.console.type = Console
+appender.console.name = CONSOLE
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
+
+# Root logger level
+rootLogger.level = warn
+# Root logger referring to console appender
+rootLogger.appenderRef.stdout.ref = CONSOLE
+
+# Set the default spark-shell log level to WARN. When running the spark-shell,
the
+# log level for this class is used to overwrite the root logger's log level,
so that
+# the user can have different defaults for the shell and regular Spark apps.
+logger.apache_spark_repl.name = org.apache.spark.repl.Main
+logger.apache_spark_repl.level = warn
+# Set logging of integration testsuite to INFO level
+logger.hudi_integ.name = org.apache.hudi.integ.testsuite
+logger.hudi_integ.level = info
+# Settings to quiet third party logs that are too verbose
+logger.apache_spark_jetty.name = org.spark_project.jetty
+logger.apache_spark_jetty.level = warn
+logger.apache_spark_jett_lifecycle.name =
org.spark_project.jetty.util.component.AbstractLifeCycle
Review Comment:
I find it really odd that we need to now come up with the name for this
loggers, instead of just identifying them by the package-name
##########
hudi-client/hudi-client-common/src/main/resources/log4j2.properties:
##########
@@ -15,9 +15,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
###
-log4j.rootLogger=INFO, A1
-# A1 is set to be a ConsoleAppender.
-log4j.appender.A1=org.apache.log4j.ConsoleAppender
-# A1 uses PatternLayout.
-log4j.appender.A1.layout=org.apache.log4j.PatternLayout
-log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
+status = warn
+name = HudiConsoleLog
Review Comment:
We need to clarify why we have this config in production (not test)
resources.
If we're previously trying to influence configuration of users of the Spark
client, then we can't upgrade this one to log4j2 since there are still clients
using older Spark versions using log4j1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
