[jira] [Commented] (HADOOP-12848) Configuration.ParsedTimeDuration.unitFor(String) should do more careful parsing
[ https://issues.apache.org/jira/browse/HADOOP-12848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170256#comment-15170256 ] Hari Shreedharan commented on HADOOP-12848: --- Hence proved, cows == pigs > Configuration.ParsedTimeDuration.unitFor(String) should do more careful > parsing > --- > > Key: HADOOP-12848 > URL: https://issues.apache.org/jira/browse/HADOOP-12848 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > The unit parsing code is very loosey-goosey. For example, "2 pigs" will > parse as 2 seconds. "3 hams" will parse as 3 milliseconds. We might want to > tighten that up a bit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-10591) Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed
Hari Shreedharan created HADOOP-10591: - Summary: Compression codecs must used pooled direct buffers or deallocate direct buffers when stream is closed Key: HADOOP-10591 URL: https://issues.apache.org/jira/browse/HADOOP-10591 Project: Hadoop Common Issue Type: Bug Reporter: Hari Shreedharan Currently direct buffers allocated by compression codecs like Gzip (which allocates 2 direct buffers per instance) are not deallocated when the stream is closed. Eventually for long running processes which create a huge number of files, these direct buffers are left hanging till a full gc, which may or may not happen in a reasonable amount of time - especially if the process does not use a whole lot of heap. Either these buffers should be pooled or they should be deallocated when the stream is closed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508592#comment-13508592
]
Hari Shreedharan commented on HADOOP-9107:
--
Karthik, Steve - makes complete sense.
Hadoop IPC client eats InterruptedException and sets interrupt on the thread
which is not documented
Key: HADOOP-9107
URL: https://issues.apache.org/jira/browse/HADOOP-9107
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Hari Shreedharan
This code in Client.java looks fishy:
{code}
public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId) throws InterruptedException, IOException {
Call call = new Call(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call);
connection.sendParam(call); // send the parameter
boolean interrupted = false;
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
// save the fact that we were interrupted
interrupted = true;
}
}
if (interrupted) {
// set the interrupt flag now that we are done waiting
Thread.currentThread().interrupt();
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResult();
}
}
}
{code}
Blocking calls are expected to throw InterruptedException if that is
interrupted. Also it seems like this method waits on the call objects even if
it is interrupted. Currently, this method does not throw an
InterruptedException, nor is it documented that this method interrupts the
thread calling it. If it is interrupted, this method should still throw
InterruptedException, it should not matter if the call was successful or not.
This is a major issue for clients which do not call this directly, but call
HDFS client API methods to write to HDFS, which may be interrupted by the
client due to timeouts, but does not throw InterruptedException. Any HDFS
client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507660#comment-13507660
]
Hari Shreedharan commented on HADOOP-9107:
--
I agree that both are pretty similar, but I think we still need to do the
cleanup I am proposing here right?
Hadoop IPC client eats InterruptedException and sets interrupt on the thread
which is not documented
Key: HADOOP-9107
URL: https://issues.apache.org/jira/browse/HADOOP-9107
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Hari Shreedharan
This code in Client.java looks fishy:
{code}
public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId) throws InterruptedException, IOException {
Call call = new Call(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call);
connection.sendParam(call); // send the parameter
boolean interrupted = false;
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
// save the fact that we were interrupted
interrupted = true;
}
}
if (interrupted) {
// set the interrupt flag now that we are done waiting
Thread.currentThread().interrupt();
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResult();
}
}
}
{code}
Blocking calls are expected to throw InterruptedException if that is
interrupted. Also it seems like this method waits on the call objects even if
it is interrupted. Currently, this method does not throw an
InterruptedException, nor is it documented that this method interrupts the
thread calling it. If it is interrupted, this method should still throw
InterruptedException, it should not matter if the call was successful or not.
This is a major issue for clients which do not call this directly, but call
HDFS client API methods to write to HDFS, which may be interrupted by the
client due to timeouts, but does not throw InterruptedException. Any HDFS
client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
Hari Shreedharan created HADOOP-9107:
Summary: Hadoop IPC client eats InterruptedException and sets
interrupt on the thread which is not documented
Key: HADOOP-9107
URL: https://issues.apache.org/jira/browse/HADOOP-9107
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 2.0.2-alpha
Reporter: Hari Shreedharan
This code in Client.java looks fishy:
{code}
public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId) throws InterruptedException, IOException {
Call call = new Call(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call);
connection.sendParam(call); // send the parameter
boolean interrupted = false;
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
// save the fact that we were interrupted
interrupted = true;
}
}
if (interrupted) {
// set the interrupt flag now that we are done waiting
Thread.currentThread().interrupt();
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResult();
}
}
}
{code}
Blocking calls are expected to throw InterruptedException if that is
interrupted. Also it seems like this method waits on the call objects even if
it is interrupted. Currently, this method does not throw an
InterruptedException, nor is it documented that this method interrupts the
thread calling it. If it is interrupted, this method should still throw
InterruptedException, it should not matter if the call was successful or not.
This is a major issue for clients which do not call this directly, but call
HDFS client API methods to write to HDFS, which may be interrupted by the
client due to timeouts, but does not throw InterruptedException. Any HDFS
client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hari Shreedharan updated HADOOP-9107:
-
Affects Version/s: 1.1.0
Hadoop IPC client eats InterruptedException and sets interrupt on the thread
which is not documented
Key: HADOOP-9107
URL: https://issues.apache.org/jira/browse/HADOOP-9107
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Hari Shreedharan
This code in Client.java looks fishy:
{code}
public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId) throws InterruptedException, IOException {
Call call = new Call(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call);
connection.sendParam(call); // send the parameter
boolean interrupted = false;
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
// save the fact that we were interrupted
interrupted = true;
}
}
if (interrupted) {
// set the interrupt flag now that we are done waiting
Thread.currentThread().interrupt();
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResult();
}
}
}
{code}
Blocking calls are expected to throw InterruptedException if that is
interrupted. Also it seems like this method waits on the call objects even if
it is interrupted. Currently, this method does not throw an
InterruptedException, nor is it documented that this method interrupts the
thread calling it. If it is interrupted, this method should still throw
InterruptedException, it should not matter if the call was successful or not.
This is a major issue for clients which do not call this directly, but call
HDFS client API methods to write to HDFS, which may be interrupted by the
client due to timeouts, but does not throw InterruptedException. Any HDFS
client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506980#comment-13506980
]
Hari Shreedharan commented on HADOOP-9107:
--
(1) is insufficient since clients often do not directly call this method. I
believe that if this method gets interrupted:
* Clean up the call object - seems like some clean up is required in the
Connection object.
* throw InterruptedException, regardless of whether the calls complete
successfully or not.
Hadoop IPC client eats InterruptedException and sets interrupt on the thread
which is not documented
Key: HADOOP-9107
URL: https://issues.apache.org/jira/browse/HADOOP-9107
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Hari Shreedharan
This code in Client.java looks fishy:
{code}
public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId) throws InterruptedException, IOException {
Call call = new Call(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call);
connection.sendParam(call); // send the parameter
boolean interrupted = false;
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
// save the fact that we were interrupted
interrupted = true;
}
}
if (interrupted) {
// set the interrupt flag now that we are done waiting
Thread.currentThread().interrupt();
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResult();
}
}
}
{code}
Blocking calls are expected to throw InterruptedException if that is
interrupted. Also it seems like this method waits on the call objects even if
it is interrupted. Currently, this method does not throw an
InterruptedException, nor is it documented that this method interrupts the
thread calling it. If it is interrupted, this method should still throw
InterruptedException, it should not matter if the call was successful or not.
This is a major issue for clients which do not call this directly, but call
HDFS client API methods to write to HDFS, which may be interrupted by the
client due to timeouts, but does not throw InterruptedException. Any HDFS
client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506983#comment-13506983
]
Hari Shreedharan commented on HADOOP-9107:
--
To ensure that the real client that calls this should know that the call was
interrupted, rather than forcing it to check the thread's interrupt flag.
Hadoop IPC client eats InterruptedException and sets interrupt on the thread
which is not documented
Key: HADOOP-9107
URL: https://issues.apache.org/jira/browse/HADOOP-9107
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Hari Shreedharan
This code in Client.java looks fishy:
{code}
public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId) throws InterruptedException, IOException {
Call call = new Call(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call);
connection.sendParam(call); // send the parameter
boolean interrupted = false;
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
// save the fact that we were interrupted
interrupted = true;
}
}
if (interrupted) {
// set the interrupt flag now that we are done waiting
Thread.currentThread().interrupt();
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResult();
}
}
}
{code}
Blocking calls are expected to throw InterruptedException if that is
interrupted. Also it seems like this method waits on the call objects even if
it is interrupted. Currently, this method does not throw an
InterruptedException, nor is it documented that this method interrupts the
thread calling it. If it is interrupted, this method should still throw
InterruptedException, it should not matter if the call was successful or not.
This is a major issue for clients which do not call this directly, but call
HDFS client API methods to write to HDFS, which may be interrupted by the
client due to timeouts, but does not throw InterruptedException. Any HDFS
client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9107) Hadoop IPC client eats InterruptedException and sets interrupt on the thread which is not documented
[
https://issues.apache.org/jira/browse/HADOOP-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507081#comment-13507081
]
Hari Shreedharan commented on HADOOP-9107:
--
My take on what should really happen in the catch block:
* call.setException()
* Remove call from the calls table.
* In the receiveResponse method, check if calls.get(callId) returns null before
proceeding.
* throw the InterruptedException (or wrap it and then throw), so client code
can know something went wrong and the call failed.
Hadoop IPC client eats InterruptedException and sets interrupt on the thread
which is not documented
Key: HADOOP-9107
URL: https://issues.apache.org/jira/browse/HADOOP-9107
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 1.1.0, 2.0.2-alpha
Reporter: Hari Shreedharan
This code in Client.java looks fishy:
{code}
public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
ConnectionId remoteId) throws InterruptedException, IOException {
Call call = new Call(rpcKind, rpcRequest);
Connection connection = getConnection(remoteId, call);
connection.sendParam(call); // send the parameter
boolean interrupted = false;
synchronized (call) {
while (!call.done) {
try {
call.wait(); // wait for the result
} catch (InterruptedException ie) {
// save the fact that we were interrupted
interrupted = true;
}
}
if (interrupted) {
// set the interrupt flag now that we are done waiting
Thread.currentThread().interrupt();
}
if (call.error != null) {
if (call.error instanceof RemoteException) {
call.error.fillInStackTrace();
throw call.error;
} else { // local exception
InetSocketAddress address = connection.getRemoteAddress();
throw NetUtils.wrapException(address.getHostName(),
address.getPort(),
NetUtils.getHostname(),
0,
call.error);
}
} else {
return call.getRpcResult();
}
}
}
{code}
Blocking calls are expected to throw InterruptedException if that is
interrupted. Also it seems like this method waits on the call objects even if
it is interrupted. Currently, this method does not throw an
InterruptedException, nor is it documented that this method interrupts the
thread calling it. If it is interrupted, this method should still throw
InterruptedException, it should not matter if the call was successful or not.
This is a major issue for clients which do not call this directly, but call
HDFS client API methods to write to HDFS, which may be interrupted by the
client due to timeouts, but does not throw InterruptedException. Any HDFS
client calls can interrupt the thread but it is not documented anywhere.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
