[jira] [Updated] (HADOOP-15429) unsynchronized index causes DataInputByteBuffer$Buffer.read hangs
[
https://issues.apache.org/jira/browse/HADOOP-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated HADOOP-15429:
--
Affects Version/s: 2.5.0
> unsynchronized index causes DataInputByteBuffer$Buffer.read hangs
> -
>
> Key: HADOOP-15429
> URL: https://issues.apache.org/jira/browse/HADOOP-15429
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.23.0, 2.5.0
>Reporter: John Doe
>Priority: Minor
>
> In DataInputByteBuffer$Buffer class, the fields bidx and buffers, etc are
> unsynchronized when used in read() and reset() function. In certain
> circumstances, e.g., the reset() is invoked in a loop, the unsynchronized
> bidx and buffers can trigger a concurrency bug.
> Here is the code snippet.
> {code:java}
> ByteBuffer[] buffers = new ByteBuffer[0];
> int bidx, pos, length;
> @Override
> public int read(byte[] b, int off, int len) {
> if (bidx >= buffers.length) {
> return -1;
> }
> int cur = 0;
> do {
> int rem = Math.min(len, buffers[bidx].remaining());
> buffers[bidx].get(b, off, rem);
> cur += rem;
> off += rem;
> len -= rem;
> } while (len > 0 && ++bidx < buffers.length); //bidx is unsynchronized
> pos += cur;
> return cur;
> }
> public void reset(ByteBuffer[] buffers) {//if one thread keeps calling
> reset() in a loop
> bidx = pos = length = 0;
> this.buffers = buffers;
> for (ByteBuffer b : buffers) {
> length += b.remaining();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15429) unsynchronized index causes DataInputByteBuffer$Buffer.read hangs
[
https://issues.apache.org/jira/browse/HADOOP-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated HADOOP-15429:
--
Description:
In DataInputByteBuffer$Buffer class, the fields bidx and buffers, etc are
unsynchronized when used in read() and reset() function. In certain
circumstances, e.g., the reset() is invoked in a loop, the unsynchronized bidx
and buffers can trigger a concurrency bug.
Here is the code snippet.
{code:java}
ByteBuffer[] buffers = new ByteBuffer[0];
int bidx, pos, length;
@Override
public int read(byte[] b, int off, int len) {
if (bidx >= buffers.length) {
return -1;
}
int cur = 0;
do {
int rem = Math.min(len, buffers[bidx].remaining());
buffers[bidx].get(b, off, rem);
cur += rem;
off += rem;
len -= rem;
} while (len > 0 && ++bidx < buffers.length); //bidx is unsynchronized
pos += cur;
return cur;
}
public void reset(ByteBuffer[] buffers) {//if one thread keeps calling
reset() in a loop
bidx = pos = length = 0;
this.buffers = buffers;
for (ByteBuffer b : buffers) {
length += b.remaining();
}
}
{code}
was:
In DataInputByteBuffer$Buffer class, the fields bidx and buffers, etc are
unsynchronized when used in read() and reset() function. In certain
circumstances, e.g., the reset() is invoked in a loop, the unsynchronized bidx
and buffers triggers a concurrency bug.
Here is the code snippet.
{code:java}
ByteBuffer[] buffers = new ByteBuffer[0];
int bidx, pos, length;
@Override
public int read(byte[] b, int off, int len) {
if (bidx >= buffers.length) {
return -1;
}
int cur = 0;
do {
int rem = Math.min(len, buffers[bidx].remaining());
buffers[bidx].get(b, off, rem);
cur += rem;
off += rem;
len -= rem;
} while (len > 0 && ++bidx < buffers.length); //bidx is unsynchronized
pos += cur;
return cur;
}
public void reset(ByteBuffer[] buffers) {//if one thread keeps calling
reset() in a loop
bidx = pos = length = 0;
this.buffers = buffers;
for (ByteBuffer b : buffers) {
length += b.remaining();
}
}
{code}
> unsynchronized index causes DataInputByteBuffer$Buffer.read hangs
> -
>
> Key: HADOOP-15429
> URL: https://issues.apache.org/jira/browse/HADOOP-15429
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.23.0
>Reporter: John Doe
>Priority: Minor
>
> In DataInputByteBuffer$Buffer class, the fields bidx and buffers, etc are
> unsynchronized when used in read() and reset() function. In certain
> circumstances, e.g., the reset() is invoked in a loop, the unsynchronized
> bidx and buffers can trigger a concurrency bug.
> Here is the code snippet.
> {code:java}
> ByteBuffer[] buffers = new ByteBuffer[0];
> int bidx, pos, length;
> @Override
> public int read(byte[] b, int off, int len) {
> if (bidx >= buffers.length) {
> return -1;
> }
> int cur = 0;
> do {
> int rem = Math.min(len, buffers[bidx].remaining());
> buffers[bidx].get(b, off, rem);
> cur += rem;
> off += rem;
> len -= rem;
> } while (len > 0 && ++bidx < buffers.length); //bidx is unsynchronized
> pos += cur;
> return cur;
> }
> public void reset(ByteBuffer[] buffers) {//if one thread keeps calling
> reset() in a loop
> bidx = pos = length = 0;
> this.buffers = buffers;
> for (ByteBuffer b : buffers) {
> length += b.remaining();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15429) unsynchronized index causes DataInputByteBuffer$Buffer.read hangs
John Doe created HADOOP-15429:
-
Summary: unsynchronized index causes
DataInputByteBuffer$Buffer.read hangs
Key: HADOOP-15429
URL: https://issues.apache.org/jira/browse/HADOOP-15429
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 0.23.0
Reporter: John Doe
In DataInputByteBuffer$Buffer class, the fields bidx and buffers, etc are
unsynchronized when used in read() and reset() function. In certain
circumstances, e.g., the reset() is invoked in a loop, the unsynchronized bidx
and buffers triggers a concurrency bug.
Here is the code snippet.
{code:java}
ByteBuffer[] buffers = new ByteBuffer[0];
int bidx, pos, length;
@Override
public int read(byte[] b, int off, int len) {
if (bidx >= buffers.length) {
return -1;
}
int cur = 0;
do {
int rem = Math.min(len, buffers[bidx].remaining());
buffers[bidx].get(b, off, rem);
cur += rem;
off += rem;
len -= rem;
} while (len > 0 && ++bidx < buffers.length); //bidx is unsynchronized
pos += cur;
return cur;
}
public void reset(ByteBuffer[] buffers) {//if one thread keeps calling
reset() in a loop
bidx = pos = length = 0;
this.buffers = buffers;
for (ByteBuffer b : buffers) {
length += b.remaining();
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15425) CopyFilesMapper.doCopyFile hangs with misconfigured sizeBuf
John Doe created HADOOP-15425:
-
Summary: CopyFilesMapper.doCopyFile hangs with misconfigured
sizeBuf
Key: HADOOP-15425
URL: https://issues.apache.org/jira/browse/HADOOP-15425
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.5.0
Reporter: John Doe
When the sizeBuf is configured to be 0, the for loop in
DistCpV1$CopyFilesMapper.doCopyFile function hangs endlessly.
This is because when the buf.size (i.e., sizeBuf) is 0, the bytesRead will
always be 0 by invoking bytesRead=in.read(buf).
Here is the code snippet.
{code:java}
sizeBuf = job.getInt("copy.buf.size", 128 * 1024); //when copy.buf.size = 0
buffer = new byte[sizeBuf];
private long doCopyFile(FileStatus srcstat, Path tmpfile, Path absdst,
Reporter reporter) throws IOException {
FSDataInputStream in = null;
FSDataOutputStream out = null;
long bytesCopied = 0L;
try {
Path srcPath = srcstat.getPath();
// open src file
in = srcPath.getFileSystem(job).open(srcPath);
reporter.incrCounter(Counter.BYTESEXPECTED, srcstat.getLen());
// open tmp file
out = create(tmpfile, reporter, srcstat);
LOG.info("Copying file " + srcPath + " of size " + srcstat.getLen() + "
bytes...");
// copy file
for(int bytesRead; (bytesRead = in.read(buffer)) >= 0; ) {
out.write(buffer, 0, bytesRead);
bytesCopied += bytesRead;
reporter.setStatus(... );
}
} finally {
checkAndClose(in);
checkAndClose(out);
}
return bytesCopied;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15424) XDR.ensureFreeSpace hangs when a corrupted bytebuffer passed into the constructor
[
https://issues.apache.org/jira/browse/HADOOP-15424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated HADOOP-15424:
--
Description:
When a corrupted bytebuffer passed into the constructor, i.e., the
bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs
endlessly.
This is because the loop stride (newCapacity) is always 0, making the loop
index (newRemaining) always less than the upper bound (size).
Here is the code snippet.
{code:java}
public XDR(ByteBuffer buf, State state) {
this.buf = buf;
this.state = state;
}
private void ensureFreeSpace(int size) {
Preconditions.checkState(state == State.WRITING);
if (buf.remaining() < size) {
int newCapacity = buf.capacity() * 2;
int newRemaining = buf.capacity() + buf.remaining();
while (newRemaining < size) {
newRemaining += newCapacity;
newCapacity *= 2;
}
ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
buf.flip();
newbuf.put(buf);
buf = newbuf;
}
}
{code}
was:
When a corrupted bytebuffer passed into the constructor, i.e., the
bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs
endlessly.
This is because the loop stride (newCapacity) is always 0, making the loop
index (newRemaining) always less than the upper bound (size).
Here is the code snippet.
{code:java}
public XDR(ByteBuffer buf, State state) {
this.buf = buf;
this.state = state;
}
private void ensureFreeSpace(int size) {
Preconditions.checkState(state == State.WRITING);
if (buf.remaining() < size) {
int newCapacity = buf.capacity() * 2;
int newRemaining = buf.capacity() + buf.remaining();
while (newRemaining < size) {
newRemaining += newCapacity;
newCapacity *= 2;
}
ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
buf.flip();
newbuf.put(buf);
buf = newbuf;
}
}
{code}
> XDR.ensureFreeSpace hangs when a corrupted bytebuffer passed into the
> constructor
> -
>
> Key: HADOOP-15424
> URL: https://issues.apache.org/jira/browse/HADOOP-15424
> Project: Hadoop Common
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.5.0
>Reporter: John Doe
>Priority: Major
>
> When a corrupted bytebuffer passed into the constructor, i.e., the
> bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs
> endlessly.
> This is because the loop stride (newCapacity) is always 0, making the loop
> index (newRemaining) always less than the upper bound (size).
> Here is the code snippet.
> {code:java}
> public XDR(ByteBuffer buf, State state) {
> this.buf = buf;
> this.state = state;
> }
> private void ensureFreeSpace(int size) {
> Preconditions.checkState(state == State.WRITING);
> if (buf.remaining() < size) {
> int newCapacity = buf.capacity() * 2;
> int newRemaining = buf.capacity() + buf.remaining();
> while (newRemaining < size) {
> newRemaining += newCapacity;
> newCapacity *= 2;
> }
> ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
> buf.flip();
> newbuf.put(buf);
> buf = newbuf;
> }
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15424) XDR.ensureFreeSpace hangs when a corrupted bytebuffer passed into the constructor
John Doe created HADOOP-15424:
-
Summary: XDR.ensureFreeSpace hangs when a corrupted bytebuffer
passed into the constructor
Key: HADOOP-15424
URL: https://issues.apache.org/jira/browse/HADOOP-15424
Project: Hadoop Common
Issue Type: Bug
Components: nfs
Affects Versions: 2.5.0
Reporter: John Doe
When a corrupted bytebuffer passed into the constructor, i.e., the
bytebuffer.capacity = 0, the while loop in XDR.ensureFreeSpace function hangs
endlessly.
This is because the loop stride (newCapacity) is always 0, making the loop
index (newRemaining) always less than the upper bound (size).
Here is the code snippet.
{code:java}
public XDR(ByteBuffer buf, State state) {
this.buf = buf;
this.state = state;
}
private void ensureFreeSpace(int size) {
Preconditions.checkState(state == State.WRITING);
if (buf.remaining() < size) {
int newCapacity = buf.capacity() * 2;
int newRemaining = buf.capacity() + buf.remaining();
while (newRemaining < size) {
newRemaining += newCapacity;
newCapacity *= 2;
}
ByteBuffer newbuf = ByteBuffer.allocate(newCapacity);
buf.flip();
newbuf.put(buf);
buf = newbuf;
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15415) copyBytes hangs when the configuration file is corrupted
[
https://issues.apache.org/jira/browse/HADOOP-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated HADOOP-15415:
--
Affects Version/s: 2.5.0
> copyBytes hangs when the configuration file is corrupted
>
>
> Key: HADOOP-15415
> URL: https://issues.apache.org/jira/browse/HADOOP-15415
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 0.23.0, 2.5.0, 3.1.0
>Reporter: John Doe
>Priority: Minor
>
> The third parameter, buffSize, is read from the configuration files or
> user-specified.
> When the configuration file is corrupted or the user configures with a wrong
> value, i.e., 0, the bytesRead will always be 0, making the while loop's
> condition always true, hanging IOUtils.copyBytes() endlessly.
> Here is the snippet of the code. There are four copyBytes in the following
> code. The 3rd and 4th copyBytes calls the 1st one. The 1st one calls the 2nd
> one. Hang happens in the while loop of the second copyBytes function.
>
> {code:java}
> //1st copyBytes
> public static void copyBytes(InputStream in, OutputStream out, int
> buffSize, boolean close)
> throws IOException {
> try {
> copyBytes(in, out, buffSize);
> if(close) {
> out.close();
> out = null;
> in.close();
> in = null;
> }
> } finally {
> if(close) {
> closeStream(out);
> closeStream(in);
> }
> }
> }
>
> //2nd copyBytes
> public static void copyBytes(InputStream in, OutputStream out, int
> buffSize)
> throws IOException {
> PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
> byte buf[] = new byte[buffSize];
> int bytesRead = in.read(buf);
> while (bytesRead >= 0) {
> out.write(buf, 0, bytesRead);
> if ((ps != null) && ps.checkError()) {
> throw new IOException("Unable to write to output stream.");
> }
> bytesRead = in.read(buf);
> }
> }
> //3rd copyBytes
> public static void copyBytes(InputStream in, OutputStream out,
> Configuration conf)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
> }
>
> //4th copyBytes
> public static void copyBytes(InputStream in, OutputStream out,
> Configuration conf, boolean close)
> throws IOException {
> copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), close);
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15417) s3: retrieveBlock hangs when the configuration file is corrupted
[
https://issues.apache.org/jira/browse/HADOOP-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated HADOOP-15417:
--
Component/s: common
> s3: retrieveBlock hangs when the configuration file is corrupted
>
>
> Key: HADOOP-15417
> URL: https://issues.apache.org/jira/browse/HADOOP-15417
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/s3
>Affects Versions: 0.23.0, 2.5.0
>Reporter: John Doe
>Priority: Major
>
> The bufferSize is read from the configuration files.
> When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will
> always be 0, making the while loop's condition always true, hanging
> Jets3tFileSystemStore.retrieveBlock() endlessly.
> Here is the snippet of the code.
> {code:java}
> private int bufferSize;
> this.bufferSize = conf.getInt(
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY,
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
> public File retrieveBlock(Block block, long byteRangeStart)
> throws IOException {
> File fileBlock = null;
> InputStream in = null;
> OutputStream out = null;
> try {
> fileBlock = newBackupFile();
> in = get(blockToKey(block), byteRangeStart);
> out = new BufferedOutputStream(new FileOutputStream(fileBlock));
> byte[] buf = new byte[bufferSize];
> int numRead;
> while ((numRead = in.read(buf)) >= 0) {
> out.write(buf, 0, numRead);
> }
> return fileBlock;
> } catch (IOException e) {
> ...
> } finally {
> ...
> }
> }
> {code}
> Similar case:
> [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15417) s3: retrieveBlock hangs when the configuration file is corrupted
[
https://issues.apache.org/jira/browse/HADOOP-15417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated HADOOP-15417:
--
Affects Version/s: 2.5.0
> s3: retrieveBlock hangs when the configuration file is corrupted
>
>
> Key: HADOOP-15417
> URL: https://issues.apache.org/jira/browse/HADOOP-15417
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs/s3
>Affects Versions: 0.23.0, 2.5.0
>Reporter: John Doe
>Priority: Major
>
> The bufferSize is read from the configuration files.
> When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will
> always be 0, making the while loop's condition always true, hanging
> Jets3tFileSystemStore.retrieveBlock() endlessly.
> Here is the snippet of the code.
> {code:java}
> private int bufferSize;
> this.bufferSize = conf.getInt(
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY,
> S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
> public File retrieveBlock(Block block, long byteRangeStart)
> throws IOException {
> File fileBlock = null;
> InputStream in = null;
> OutputStream out = null;
> try {
> fileBlock = newBackupFile();
> in = get(blockToKey(block), byteRangeStart);
> out = new BufferedOutputStream(new FileOutputStream(fileBlock));
> byte[] buf = new byte[bufferSize];
> int numRead;
> while ((numRead = in.read(buf)) >= 0) {
> out.write(buf, 0, numRead);
> }
> return fileBlock;
> } catch (IOException e) {
> ...
> } finally {
> ...
> }
> }
> {code}
> Similar case:
> [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15417) retrieveBlock hangs when the configuration file is corrupted
John Doe created HADOOP-15417:
-
Summary: retrieveBlock hangs when the configuration file is
corrupted
Key: HADOOP-15417
URL: https://issues.apache.org/jira/browse/HADOOP-15417
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 0.23.0
Reporter: John Doe
The bufferSize is read from the configuration files.
When the configuration file is corrupted, i.e.,bufferSize=0, the numRead will
always be 0, making the while loop's condition always true, hanging
Jets3tFileSystemStore.retrieveBlock() endlessly.
Here is the snippet of the code.
{code:java}
private int bufferSize;
this.bufferSize = conf.getInt(
S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_KEY,
S3FileSystemConfigKeys.S3_STREAM_BUFFER_SIZE_DEFAULT);
public File retrieveBlock(Block block, long byteRangeStart)
throws IOException {
File fileBlock = null;
InputStream in = null;
OutputStream out = null;
try {
fileBlock = newBackupFile();
in = get(blockToKey(block), byteRangeStart);
out = new BufferedOutputStream(new FileOutputStream(fileBlock));
byte[] buf = new byte[bufferSize];
int numRead;
while ((numRead = in.read(buf)) >= 0) {
out.write(buf, 0, numRead);
}
return fileBlock;
} catch (IOException e) {
...
} finally {
...
}
}
{code}
Similar case: [Hadoop-15415|https://issues.apache.org/jira/browse/HADOOP-15415].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15415) copyBytes hangs when the configuration file is corrupted
John Doe created HADOOP-15415:
-
Summary: copyBytes hangs when the configuration file is corrupted
Key: HADOOP-15415
URL: https://issues.apache.org/jira/browse/HADOOP-15415
Project: Hadoop Common
Issue Type: Bug
Components: common
Affects Versions: 0.23.0
Reporter: John Doe
The third parameter, buffSize, is read from the configuration files or
user-specified.
When the configuration file is corrupted or the user configures with a wrong
value, i.e., 0, the bytesRead will always be 0, making the while loop's
condition always true, hanging IOUtils.copyBytes() endlessly.
Here is the snippet of the code. There are four copyBytes in the following
code. The 3rd and 4th copyBytes calls the 1st one. The 1st one calls the 2nd
one. Hang happens in the while loop of the second copyBytes function.
{code:java}
//1st copyBytes
public static void copyBytes(InputStream in, OutputStream out, int buffSize,
boolean close)
throws IOException {
try {
copyBytes(in, out, buffSize);
if(close) {
out.close();
out = null;
in.close();
in = null;
}
} finally {
if(close) {
closeStream(out);
closeStream(in);
}
}
}
//2nd copyBytes
public static void copyBytes(InputStream in, OutputStream out, int buffSize)
throws IOException {
PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
byte buf[] = new byte[buffSize];
int bytesRead = in.read(buf);
while (bytesRead >= 0) {
out.write(buf, 0, bytesRead);
if ((ps != null) && ps.checkError()) {
throw new IOException("Unable to write to output stream.");
}
bytesRead = in.read(buf);
}
}
//3rd copyBytes
public static void copyBytes(InputStream in, OutputStream out, Configuration
conf)
throws IOException {
copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), true);
}
//4th copyBytes
public static void copyBytes(InputStream in, OutputStream out, Configuration
conf, boolean close)
throws IOException {
copyBytes(in, out, conf.getInt("io.file.buffer.size", 4096), close);
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15088) BufferedInputStream.skip function can return 0 when the file is corrupted, causing an infinite loop
[
https://issues.apache.org/jira/browse/HADOOP-15088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Doe updated HADOOP-15088:
--
Description:
When a file is corrupted, the skip function can return 0, causing an infinite
loop.
Here is the code:
{code:java}
private boolean slowReadUntilMatch(Pattern markPattern, boolean includePat,
DataOutputBuffer outBufOrNull) throws
IOException {
...
for (long skiplen = endPos; skiplen > 0; ) {
skiplen -= bin_.skip(skiplen);
}
...
}
{code}
Similar bugs are
[Hadoop-8614|https://issues.apache.org/jira/browse/HADOOP-8614],
[Yarn-2905|https://issues.apache.org/jira/browse/YARN-2905],
[Yarn-163|https://issues.apache.org/jira/browse/YARN-163],
[MAPREDUCE-6990|https://issues.apache.org/jira/browse/MAPREDUCE-6990]
was:
When a file is corrupted, the skip function can return 0, causing an infinite
loop.
Here is the code:
{code:java}
private boolean slowReadUntilMatch(Pattern markPattern, boolean includePat,
DataOutputBuffer outBufOrNull) throws
IOException {
...
for (long skiplen = endPos; skiplen > 0; ) {
skiplen -= bin_.skip(skiplen);
}
...
}
{code}
> BufferedInputStream.skip function can return 0 when the file is corrupted,
> causing an infinite loop
> ---
>
> Key: HADOOP-15088
> URL: https://issues.apache.org/jira/browse/HADOOP-15088
> Project: Hadoop Common
> Issue Type: Bug
> Components: streaming
>Affects Versions: 2.5.0
>Reporter: John Doe
>
> When a file is corrupted, the skip function can return 0, causing an infinite
> loop.
> Here is the code:
> {code:java}
> private boolean slowReadUntilMatch(Pattern markPattern, boolean includePat,
> DataOutputBuffer outBufOrNull) throws
> IOException {
> ...
> for (long skiplen = endPos; skiplen > 0; ) {
> skiplen -= bin_.skip(skiplen);
> }
> ...
> }
> {code}
> Similar bugs are
> [Hadoop-8614|https://issues.apache.org/jira/browse/HADOOP-8614],
> [Yarn-2905|https://issues.apache.org/jira/browse/YARN-2905],
> [Yarn-163|https://issues.apache.org/jira/browse/YARN-163],
> [MAPREDUCE-6990|https://issues.apache.org/jira/browse/MAPREDUCE-6990]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15088) BufferedInputStream.skip function can return 0 when the file is corrupted, causing an infinite loop
John Doe created HADOOP-15088:
-
Summary: BufferedInputStream.skip function can return 0 when the
file is corrupted, causing an infinite loop
Key: HADOOP-15088
URL: https://issues.apache.org/jira/browse/HADOOP-15088
Project: Hadoop Common
Issue Type: Bug
Components: streaming
Affects Versions: 2.5.0
Reporter: John Doe
When a file is corrupted, the skip function can return 0, causing an infinite
loop.
Here is the code:
{code:java}
private boolean slowReadUntilMatch(Pattern markPattern, boolean includePat,
DataOutputBuffer outBufOrNull) throws
IOException {
...
for (long skiplen = endPos; skiplen > 0; ) {
skiplen -= bin_.skip(skiplen);
}
...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
