[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922923#comment-16922923
]
leesf commented on HUDI-233:
[~vinoth]
I have already started testing PR#873, but my environment has encountered some
problems. After testing, I will feedback the problems I encountered.
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922634#comment-16922634
]
Vinoth Chandar commented on HUDI-233:
-
Okay. Then, may be we should expand scope of the ticket to "Redo log statements
with slf4j" . On starting execution, I would still love it if we first finish
up HUDI-159 (any help with testing would be great!) and get to this. That way
we can start from a baseline that we understand fully. Thoughts?
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922329#comment-16922329
]
leesf commented on HUDI-233:
> If you can shed more light on how we would package slf4j, that would be
> great? From your description, may be we can skip bundling slf4j for hive and
> spark, but include it for presto?
Yes. we need to move back slf4j again for logging, Exclude slf4j for hive/spark
and include it for presto.
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921183#comment-16921183
]
leesf commented on HUDI-233:
[~vinoth], With log4j dependency only, one solution is to use String.format
for logging, such as _log.info(String.format("%s is a test", "this"));_ but not
very convenient.
And after a quick look to spark and hive. Spark has slf4j and log4j
dependencies in pom.xml([https://github.com/apache/spark/blob/master/pom.xml]),
and use _Logger LOG = LoggerFactory#getLogger_ for logging. Also hive uses
slf4j and log4j for logging, they are defined in
pom.xml([https://github.com/apache/hive/blob/master/pom.xml]). So do we have to
remove the slf4j dependency?
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921110#comment-16921110
]
Vinoth Chandar commented on HUDI-233:
-
Just got the PR up for new bundling. BTW sure log4j does not support this?
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920180#comment-16920180
]
Vinoth Chandar commented on HUDI-233:
-
[~xleesf] I am working on thinning the amount of dependencies we have in Hudi
to reduce probability of clashing with external systems/runtime environments it
integrates with. One of them I just removed is slf4j.. While I am not against
the specific framework (I obviously pretty good), I feel we need to be lean
towards having our helper wrappers for this sort of stuff vs bringing in new
dependencies just for this..
Can we revisit this in a week or two when I have better idea about how this
will affect shading/bundling? thoughts?
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919970#comment-16919970
]
leesf commented on HUDI-233:
[~vinoth] I would like to use slf4j along with log4j to solve this, the
dependencies are defined in pom.xml. As for conflicts, we could exclude the
dependencies. Any other concerns?
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HUDI-233) Redo log statements using {} variable substitution
[
https://issues.apache.org/jira/browse/HUDI-233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919923#comment-16919923
]
leesf commented on HUDI-233:
Big +1 for this logging format since it can improve performance of logging with
variable substitution. I will check the old logging format in the whole project
and start the progress.
> Redo log statements using {} variable substitution
> --
>
> Key: HUDI-233
> URL: https://issues.apache.org/jira/browse/HUDI-233
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: newbie, Performance
>Affects Versions: 0.5.0
>Reporter: Vinoth Chandar
>Assignee: leesf
>Priority: Major
>
> Currently we are not employing variable substitution aggresively in the
> project. ala
> {code:java}
> LogManager.getLogger(SomeName.class.getName()).info("Message: {}, Detail:
> {}", message, detail);
> {code}
> This can improve performance since the string concatenation is deferrable to
> when the logging is actually in effect.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
