[
https://issues.apache.org/jira/browse/HIVE-25970?focusedWorklogId=732215&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-732215
]
ASF GitHub Bot logged work on HIVE-25970:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Feb/22 10:47
Start Date: 24/Feb/22 10:47
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #3048:
URL: https://github.com/apache/hive/pull/3048#discussion_r813760610
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/log/HushableRandomAccessFileAppender.java
##########
@@ -51,17 +47,6 @@
public final class HushableRandomAccessFileAppender extends
AbstractOutputStreamAppender<RandomAccessFileManager> {
- private static final LoadingCache<String, String> CLOSED_FILES =
Review comment:
The `TestOperationLoggingAPIWithTez` verifies that certain messages
appear in the operation logs so it is a good guard against regressions. Apart
from adding more log messages to assert in `TestOperationLoggingAPIWithTez` I
don't have better ideas for ensuring that it will not break in the future.
I guess HIVE-22753 didn't add any tests cause it was targeting a memory leak
and these stuff are not always easy to unit test. In that case I guess it would
require executing hundreds or thousands of queries till the problem shows up.
If you have ideas on how we could improve these tests, I would be happy to
make any changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 732215)
Time Spent: 0.5h (was: 20m)
> Missing messages in HS2 operation logs
> --------------------------------------
>
> Key: HIVE-25970
> URL: https://issues.apache.org/jira/browse/HIVE-25970
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Stamatis Zampetakis
> Assignee: Stamatis Zampetakis
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> After HIVE-22753 & HIVE-24590, with some unlucky timing of events, operation
> log messages can get lost and never appear in the appropriate files.
> The changes in HIVE-22753 will prevent a {{HushableRandomAccessFileAppender}}
> from being created if the latter refers to a file that has been closed in the
> last second. Preventing the creation of the appender also means that the
> message which triggered the creation will be lost forever. In fact any
> message (for the same query) that comes in the interval of 1 second will be
> lost forever.
> Before HIVE-24590 the appender/file was closed only once (explicitly by HS2)
> and thus the problem may be very hard to notice in practice. However, with
> the arrival of HIVE-24590 appenders may close much more frequently (and not
> via HS2) making the issue reproducible rather easily. It suffices to set
> _hive.server2.operation.log.purgePolicy.timeToLive_ property very low and
> check the operation logs.
> The problem was discovered by investigating some intermittent failures in
> operation logging tests (e.g., TestOperationLoggingAPIWithTez).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
