[jira] [Resolved] (HDFS-10194) FSDataOutputStream.write() allocates new byte buffer on each operation

2016-03-22 Thread Vladimir Rodionov (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Rodionov resolved HDFS-10194.
--
Resolution: Invalid

HDFS-7276 provides ByteArrayManager which is off, by default. Enabling this 
feature resolves the issue.

> FSDataOutputStream.write() allocates new byte buffer on each operation
> --
>
> Key: HDFS-10194
> URL: https://issues.apache.org/jira/browse/HDFS-10194
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 2.7.1
>Reporter: Vladimir Rodionov
>
> This is the code:
> {code}
>  private DFSPacket createPacket(int packetSize, int chunksPerPkt, long 
> offsetInBlock, long seqno, boolean lastPacketInBlock) throws 
> InterruptedIOException {
>  final byte[] buf;
>  final int bufferSize = PacketHeader.PKT_MAX_HEADER_LEN +   packetSize;
>  
>  try {
>buf = byteArrayManager.newByteArray(bufferSize);
>  } catch (InterruptedException ie) {
>final InterruptedIOException iioe = new InterruptedIOException(
>"seqno=" + seqno);
>iioe.initCause(ie);
>throw iioe;
>  }
>  
>  return new DFSPacket(buf, chunksPerPkt, offsetInBlock, seqno,
>   getChecksumSize(), lastPacketInBlock);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10194) FSDataOutputStream.write() allocates new byte buffer on each operation

2016-03-22 Thread Vladimir Rodionov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207454#comment-15207454
 ] 

Vladimir Rodionov commented on HDFS-10194:
--

OK, it seems this is HDFS-7276 related (memory management is disabled, by 
default). I will close this one once I confirm that ByteArrayManager works.

> FSDataOutputStream.write() allocates new byte buffer on each operation
> --
>
> Key: HDFS-10194
> URL: https://issues.apache.org/jira/browse/HDFS-10194
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 2.7.1
>Reporter: Vladimir Rodionov
>
> This is the code:
> {code}
>  private DFSPacket createPacket(int packetSize, int chunksPerPkt, long 
> offsetInBlock, long seqno, boolean lastPacketInBlock) throws 
> InterruptedIOException {
>  final byte[] buf;
>  final int bufferSize = PacketHeader.PKT_MAX_HEADER_LEN +   packetSize;
>  
>  try {
>buf = byteArrayManager.newByteArray(bufferSize);
>  } catch (InterruptedException ie) {
>final InterruptedIOException iioe = new InterruptedIOException(
>"seqno=" + seqno);
>iioe.initCause(ie);
>throw iioe;
>  }
>  
>  return new DFSPacket(buf, chunksPerPkt, offsetInBlock, seqno,
>   getChecksumSize(), lastPacketInBlock);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HDFS-10194) FSDataOutputStream.write() allocates new byte buffer on each operation

2016-03-22 Thread Vladimir Rodionov (JIRA) 

[ 
https://issues.apache.org/jira/browse/HDFS-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207395#comment-15207395
 ] 

Vladimir Rodionov commented on HDFS-10194:
--

I will try to explain how this affects HBase.

HBase well-known issue is bad behavior under compaction stress. When HBase 
Compactor writes new file it uses, of course DFS (HDFS) write API. On every 1MB 
written to hdfs, HBase RS JVM allocates 1MB buffer in Eden space. If HBase 
writes 100MB sec - its 100MB sec in Eden space. Young GC get triggered more 
frequently, which results in false object promotion to tenured space, which 
results eventually in long full GC pauses, which, in turn, sometimes results in 
RS crash. 

This is for CMS. 

> FSDataOutputStream.write() allocates new byte buffer on each operation
> --
>
> Key: HDFS-10194
> URL: https://issues.apache.org/jira/browse/HDFS-10194
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 2.7.1
>Reporter: Vladimir Rodionov
>
> This is the code:
> {code}
>  private DFSPacket createPacket(int packetSize, int chunksPerPkt, long 
> offsetInBlock, long seqno, boolean lastPacketInBlock) throws 
> InterruptedIOException {
>  final byte[] buf;
>  final int bufferSize = PacketHeader.PKT_MAX_HEADER_LEN +   packetSize;
>  
>  try {
>buf = byteArrayManager.newByteArray(bufferSize);
>  } catch (InterruptedException ie) {
>final InterruptedIOException iioe = new InterruptedIOException(
>"seqno=" + seqno);
>iioe.initCause(ie);
>throw iioe;
>  }
>  
>  return new DFSPacket(buf, chunksPerPkt, offsetInBlock, seqno,
>   getChecksumSize(), lastPacketInBlock);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-10194) FSDataOutputStream.write() allocates new byte buffer on each operation

2016-03-22 Thread Vladimir Rodionov (JIRA) 
Vladimir Rodionov created HDFS-10194:


 Summary: FSDataOutputStream.write() allocates new byte buffer on 
each operation
 Key: HDFS-10194
 URL: https://issues.apache.org/jira/browse/HDFS-10194
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Affects Versions: 2.7.1
Reporter: Vladimir Rodionov


This is the code:
{code}
 private DFSPacket createPacket(int packetSize, int chunksPerPkt, long 
offsetInBlock, long seqno, boolean lastPacketInBlock) throws 
InterruptedIOException {
 final byte[] buf;
 final int bufferSize = PacketHeader.PKT_MAX_HEADER_LEN +   packetSize;
 
 try {
   buf = byteArrayManager.newByteArray(bufferSize);
 } catch (InterruptedException ie) {
   final InterruptedIOException iioe = new InterruptedIOException(
   "seqno=" + seqno);
   iioe.initCause(ie);
   throw iioe;
 }
 
 return new DFSPacket(buf, chunksPerPkt, offsetInBlock, seqno,
  getChecksumSize(), lastPacketInBlock);
}
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)