Elad Yosifon created PARQUET-2036:
-------------------------------------
Summary: implicitly defining DEBUG mode in MessageColumnIO causes
80% performance overhead
Key: PARQUET-2036
URL: https://issues.apache.org/jira/browse/PARQUET-2036
Project: Parquet
Issue Type: Bug
Affects Versions: 1.12.0, 1.10.1, 1.10.0
Reporter: Elad Yosifon
*parquet-column* jar leverages +slf4j and log4j as default logger+, neglecting
to define a log4j configuration, defaults to *DEBUG* log level.
{code:java}
public class MessageColumnIO extends GroupColumnIO {
private static final Logger LOG =
LoggerFactory.getLogger(MessageColumnIO.class);
private static final boolean DEBUG = LOG.isDebugEnabled(); // <------
}
{code}
this "magic behavior" defaults parquet library to be in DEBUG mode, without any
notification or warnings. Unfortunately, the *RecordConsumerLoggingWrapper*
implementation generates 5x performance overhead in comparison to the
*MessageColumnIORecordConsumer* implementation, causing a massive hit in
performance and wasteful server utilization.
+IMHO there are two things that could prevent such issue:+
* printing a message to STDOUT notifying about DEBUG mode being set to active.
* defaulting to *MessageColumnIORecordConsumer* implementation, and waiting
for explicit configuration to define DEBUG mode, and use
*RecordConsumerLoggingWrapper*.
In the past 2 years, this issue probably cost my company 50,000$ in excessive
cloud costs!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
