[
https://issues.apache.org/jira/browse/HDFS-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sverre Bakke updated HDFS-7941:
-------------------------------
Description:
When using SequenceFile.Writer and appending+syncing to file repeatedly, the
sync does not appear to work other than:
- once after writing headers
- when closing.
Imagine the following test case:
http://pastebin.com/Y9xysCRX
This code would append a new record every second and then immediately sync it.
One would also imagine that the file would grow for every append, however, this
does not happen.
After watching the behavior I have noticed that it only syncs the headers at
the very beginning (providing a file of 164 bytes) and then never again until
its closed. This despite it is asked to hsync() after every append.
Looking into the debug logs, this also claims the same behavior (executed the
provided code example and grepped for "sync"):
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms
This was the only time the code ran fsync throughout the entire execution.
This has been tested (with similar result) for the following deployments:
- sequencefile with no compression
- sequencefile with record compression
- sequencefile with block compression
- textfile with no compression
was:
When using SequenceFile.Writer and appending+syncing to file repeatedly, the
sync does not appear to work other than:
- once after writing headers
- when closing.
Imagine the following test case:
http://pastebin.com/Y9xysCRX
This code would append a new record every second and then immediately sync it.
One would also imagine that the file would grow for every append, however, this
does not happen.
After watching the behavior I have noticed that it only syncs the headers at
the very beginning (providing a file of 164 bytes) and then never again until
its closed. This despite it is asked to hsync() after every append.
Looking into the debug logs, this also claims the same behavior (executed the
provided code example and grepped for "sync"):
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms
This was the only time the code ran fsync throughout the entire execution.
> hsync() not working
> -------------------
>
> Key: HDFS-7941
> URL: https://issues.apache.org/jira/browse/HDFS-7941
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 2.6.0
> Environment: HDP 2.2 running on Redhat
> Reporter: Sverre Bakke
>
> When using SequenceFile.Writer and appending+syncing to file repeatedly, the
> sync does not appear to work other than:
> - once after writing headers
> - when closing.
> Imagine the following test case:
> http://pastebin.com/Y9xysCRX
> This code would append a new record every second and then immediately sync
> it. One would also imagine that the file would grow for every append,
> however, this does not happen.
> After watching the behavior I have noticed that it only syncs the headers at
> the very beginning (providing a file of 164 bytes) and then never again until
> its closed. This despite it is asked to hsync() after every append.
> Looking into the debug logs, this also claims the same behavior (executed the
> provided code example and grepped for "sync"):
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> 2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms
> This was the only time the code ran fsync throughout the entire execution.
> This has been tested (with similar result) for the following deployments:
> - sequencefile with no compression
> - sequencefile with record compression
> - sequencefile with block compression
> - textfile with no compression
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
