[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494456#comment-13494456
]
Hadoop QA commented on HADOOP-6311:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12552900/HADOOP-6311.028.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1727//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1727//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, design.txt,
> HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch,
> HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch,
> HADOOP-6311.022.patch, HADOOP-6311.023.patch, HADOOP-6311.024.patch,
> HADOOP-6311.027.patch, HADOOP-6311.028.patch, HADOOP-6311-0.patch,
> HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491852#comment-13491852
]
Hadoop QA commented on HADOOP-6311:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12552330/HADOOP-6311.027.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1713//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1713//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, design.txt,
> HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch,
> HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch,
> HADOOP-6311.022.patch, HADOOP-6311.023.patch, HADOOP-6311.024.patch,
> HADOOP-6311.027.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch,
> hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491728#comment-13491728
]
Hadoop QA commented on HADOOP-6311:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12552310/HADOOP-6311.024.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1712//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1712//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1712//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, design.txt,
> HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch,
> HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch,
> HADOOP-6311.022.patch, HADOOP-6311.023.patch, HADOOP-6311.024.patch,
> HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491217#comment-13491217
]
Hadoop QA commented on HADOOP-6311:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12552207/HADOOP-6311.023.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.net.unix.TestDomainSocket
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1710//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1710//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1710//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, design.txt,
> HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch,
> HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch,
> HADOOP-6311.022.patch, HADOOP-6311.023.patch, HADOOP-6311-0.patch,
> HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491182#comment-13491182
]
Colin Patrick McCabe commented on HADOOP-6311:
--
This latest patch is a new approach, based on the ideas we discussed here.
Basically, the idea is to add a set of generic Socket classes that implement
UNIX domain sockets.
This patch contains no code from Android. The advantages over the previous
approaches are:
* The previous approaches didn't create classes which actually inherited from
{{ServerSocket}} and {{Socket}}. This patch does. This means that
{{DomainSockets}} can be used in all of the places where {{InetSockets}} were
previously used (see the HDFS-347 patch for the application.)
* This patch has unit tests.
* We skip implementing a few things that we don't really need, like
credential-passing
* These classes are fully thread-safe.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, design.txt,
> HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch,
> HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch,
> HADOOP-6311.022.patch, HADOOP-6311.023.patch, HADOOP-6311-0.patch,
> HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477471#comment-13477471 ] Colin Patrick McCabe commented on HADOOP-6311: -- With regard to security: after further reflection, I think we will need to ask system administrators establish secure directories to hold the UNIX domain sockets. In practice, this means a directory owned by hdfs, where neither it nor any of its parent directories are vulnerable to attack. bq. ... my question is more this: let's assume that unix sockets for the data path are 3x as fast as local TCP sockets. If that's the case, then do we still get a big benefit from short-circuit? Oh, I misinterpreted. You were talking about using UNIX domain instead of TCP for data traffic. Yeah, it could be interesting. It's a time-honored way to get better performance on UNIXes. I'll do some tests if I can get the UNIX domain sockets to implement the "standard" interface (and if the resulting combination actually works.) I think there is a good chance that it will... > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, design.txt, > HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch, > HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch, > HADOOP-6311.022.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch, > hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477467#comment-13477467 ] Todd Lipcon commented on HADOOP-6311: - bq. With respect to security: there is always a possibility for a client to open a socket with the same name as the server would have used. This is similar to the problem with TCP/IP sockets of a malicious program grabbing the port before the DataNode could get it (or after the DataNode has died.) That's why secure clusters use low (privileged) ports for the data transfer protocol. bq. I don't think that on-disk format changes are that big of a deal for the short-circuit pathway. We tell old clients they can't use short-circuit reads on those files, and fix new clients to understand the new format. Agreed, just need to make sure the "deny" pathway works and ideally some kind of version number exposed. bq. TCP optimizations are pretty cool, but not when you run on RHEL6, as many folks do Maybe we should open a separate JIRA to investigate things like TCP fast open, changing TCP kernel options, etc. might be used with Hadoop in the future. There are also certain performance improvements we could do in the read and write paths on the DataNode, but again, that's out of scope for this JIRA, I think. Agreed, but my question is more this: let's assume that unix sockets for the data path are 3x as fast as local TCP sockets. If that's the case, then do we still get a big benefit from short-circuit? I think the answer is probably yes for random read, but no for sequential. The point about trying the "tcp friends" in future versions is just one potential way of evaluating this without having to write all the code for a unix socket data path. If "tcp friends" is comparable to short circuit, then unix sockets would probably also be comparable. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, design.txt, > HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch, > HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch, > HADOOP-6311.022.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch, > hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477419#comment-13477419 ] Colin Patrick McCabe commented on HADOOP-6311: -- Thanks for the comments. With respect to security: there is always a possibility for a client to open a socket with the same name as the server would have used. This is similar to the problem with TCP/IP sockets of a malicious program grabbing the port before the DataNode could get it (or after the DataNode has died.) I guess this is a problem that actually is worse with the abstract socket namespace. With path-based sockets, you can set up the path so that the permissions of the path itself prevent this attack. However, with the abstract socket namespace, there's no way to prevent another process from grabbing the port first. I agree that there are downsides to the short-circuit approach. I was very careful to maintain the ability for the server to decline to offer short-circuit local reads in my patch set. This is obviously important for our future flexibility. It might be advisable to allow this on a file-by-file basis as well. I don't think that on-disk format changes are that big of a deal for the short-circuit pathway. We tell old clients they can't use short-circuit reads on those files, and fix new clients to understand the new format. We should definitely have a way for short-circuit clients to report statistics, disk errors, etc. to the DataNode. However, let's not gate this change on features like that. They can easily be added as features later and aren't really related to the core issue of fixing local reads + security. I think I'll open a separate JIRA for that. TCP optimizations are pretty cool, but not when you run on RHEL6, as many folks do :) Maybe we should open a separate JIRA to investigate things like TCP fast open, changing TCP kernel options, etc. might be used with Hadoop in the future. There are also certain performance improvements we could do in the read and write paths on the DataNode, but again, that's out of scope for this JIRA, I think. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, design.txt, > HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch, > HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch, > HADOOP-6311.022.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch, > hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477345#comment-13477345
]
Todd Lipcon commented on HADOOP-6311:
-
Hi Colin,
Thanks for writing up the design doc. I think it probably should actually go on
HDFS-347, which is the overall feature JIRA, rather than this one, which is
about one of the implementation subtasks. But, anyway, here are some comments:
{quote}
* Portable. Hadoop supports multiple operating systems and environments,
including Linux, Solaris, and Windows.
{quote}
IMO, there is not a requirement that performance enhancements work on all
systems. It is up to the maintainers of each port to come up with the most
efficient way to do things. Though there is an active effort to get Hadoop
working on Windows, it is not yet a requirement.
So long as we maintain the current TCP-based read (which we have to, anyway,
for remote access), we'll have portability. If the Windows port doesn't
initially offer this feature, that seems acceptable to me, as they could later
add whatever mechanism makes the most sense for them.
{quote}
* High performance. If performance is compromised, there is no point to any of
this work, because clients could simply use the existing, non-short-circuit
write pathways to access data.
{quote}
Should clarify that the performance of the mechanism by which FDs are _passed_
is less important, since the client will cache the open FDs and just re-use
them for subsequent random reads against the same file (the primary use case
for this improvement). So long as the overhead of passing the FDs isn't huge,
we should be OK.
{quote}
There are other problems. How would the datanode clients and the server decide
on a socket path? If it asks every time prior to connecting, that could be
slow. If the DFSClient cached this socket path, how long should it cache it
before expiring the cache? What happens if the administrator does not properly
set up the socket path, as discussed earlier? What happens if the
administrator wants to put multiple DataNodes on the same node?
{quote}
Per above, slowness here is not a concern, since we only need to do the
socket-passing on file open. HDFS applications generally open a file once and
then perform many many reads against the same block before opening the next
block.
As for how the socket path is communicated, why not do it via an RPC? For
example, in your solution #3, we're using an RPC to communicate a cookie.
Instead of that, it can just return its abstract namespace socket name. (You
seem to propose this under solution #3 below, but here in solution #1 reject it)
Another option would be to add a new field to the
DatanodeId/DatanodeRegistration: when the client gets block locations it could
also include the socket paths.
{quote}
The response is not a path, but a 64-bit cookie. The DFSClient then connects
to the DN via a UNIX domain socket, and presents the cookie. In response, he
receives the file descriptor.
{quote}
I don't see the purpose of the cookie, still, since it adds yet another opaque
token, and requires the DN code to "publish" the file descriptor with a cookie,
and we end up with extra data structures, cached open files, cache expiration
policies, etc.
{quote}
Choice #3. Blocking FdServer versus non-blocking FdServer.
Non-blocking servers in C are somewhat more complex than blocking servers.
However, if I used a blocking server, there would be no obvious way to
determine how many threads it should use. Because it depends on how busy the
server is expected to be, only the system administrator can know ahead of time.
Additionally, many schedulers do not deal well with a large number of threads,
especially on older versions of Linux and commercial UNIX variants.
Coincidentally, these happen to be the exactly kind of systems many of our
users run.
{quote}
I don't really buy this. The socket only needs to be active long enough to pass
a single fd, which should take a few milliseconds. The number of requests for
fd-passing is based on the number of block opens, _not_ the number of reads. So
a small handful of threads should be able to handle even significant workloads
just fine. We also do fine with threads on the data xceiver path, often
configured into the hundreds or thousands.
{quote}
Another problem with blocking servers is that shutting them down can be
difficult. Since there is no time limit on blocking I/O, a message sent to the
server to terminate may take a while, or possibly forever, to be acted on.
This may seem like a trivial or unimportant problem, but it is a very real one
in unit tests. Socket receive and send timeouts can reduce the extra time
needed to shut down, but never quite eliminate it.
{quote}
Again I don't buy it, we do fine with blocking IO everywhere else.. Why is this
context different?
*Wire protocol*
The wire p
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474702#comment-13474702
]
Hadoop QA commented on HADOOP-6311:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548835/design.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1617//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, design.txt,
> HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch,
> HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch,
> HADOOP-6311.022.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch,
> hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474699#comment-13474699 ] Colin Patrick McCabe commented on HADOOP-6311: -- I posted a design document which should clarify some of this. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, design.txt, > HADOOP-6311.014.patch, HADOOP-6311.016.patch, HADOOP-6311.018.patch, > HADOOP-6311.020b.patch, HADOOP-6311.020.patch, HADOOP-6311.021.patch, > HADOOP-6311.022.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch, > hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474694#comment-13474694 ] Andy Isaacson commented on HADOOP-6311: --- Thinking about this further, it seems like the native code should be the minimal possible, and the database of which FDs are accessible should be managed in Java code. This would remove the need for the red-black tree in C, and it would make the whole patch much smaller. Todd suggested a way to completely avoid needing the cookie, as well. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch, > HADOOP-6311.020.patch, HADOOP-6311.021.patch, HADOOP-6311.022.patch, > HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474589#comment-13474589
]
Hadoop QA commented on HADOOP-6311:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548810/HADOOP-6311.022.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1614//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1614//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch,
> HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch,
> HADOOP-6311.020.patch, HADOOP-6311.021.patch, HADOOP-6311.022.patch,
> HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473787#comment-13473787 ] Colin Patrick McCabe commented on HADOOP-6311: -- bq. We need a design doc explaining how all of these bits work together. This Jira has gone on long enough that it does not serve as documentation. I think maybe a long-form comment in fd_server.c might be best? It's conceptually simple, but there are some details which might make more sense in context. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch, > HADOOP-6311.020.patch, HADOOP-6311.021.patch, HADOOP-6311-0.patch, > HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473784#comment-13473784
]
Colin Patrick McCabe commented on HADOOP-6311:
--
Thanks for the review, Andy.
bq. [red-black, splay tree implementations]
We already use the same red-black tree implementation in libhdfs. It comes
directly out of the BSD kernel and is released under a liberal license. It has
to be a tree because hash tables are unordered, and we have to expire old file
descriptors that were not claimed within a certain time period. That means
ordering by time. The trees are also much more flexible in a lot of ways.
bq. I think [ {{unixDomainSetupSockaddr}} ] can be static, right?
It's also used in fd_client.c.
bq. Put a maximum count on [RETRY_ON_EINTR]
This is basically my version of {{TEMP_FAILURE_RETRY}}, a fine macro which
happens to be a Linux-ism.
[http://www.gnu.org/software/libc/manual/html_node/Interrupted-Primitives.html#Interrupted-Primitives]
{{TEMP_FAILURE_RETRY}} doesn't have a maximum count, and as far as I know,
there's no reason why you would need one.
bq. [safe write]
I'll add the extra check for too-large write() results.
The contract is "write exactly the given amount or return an error." If we get
an error while writing to the socket, the exact amount written prior to that is
not interesting (in this case at least.)
bq. [dup / fseek issues]
I don't think it's an issue in the particular use cases we have planned (i.e.,
HDFS-347.) But it should definitely be documented.
bq. [Use /dev/urandom]
It's a Linux-ism. but I guess we could do it on that platform.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch,
> HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch,
> HADOOP-6311.020.patch, HADOOP-6311.021.patch, HADOOP-6311-0.patch,
> HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473757#comment-13473757
]
Andy Isaacson commented on HADOOP-6311:
---
High level comments:
* We need a design doc explaining how all of these bits work together. This
Jira has gone on long enough that it does not serve as documentation.
* I didn't review the red-black and splay tree implementations at all. I'm not
sure why we expect this to be big/contended enough to deserve anything beyond a
trivial hash table, which takes about 20 lines of C. (ah, I see the code comes
from \*BSD, so that's good at least. We should document where and what version
it came from for future maintainers' sanity.)
{code}
+++
hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/net/fd_server.h
...
+/**
+ * This file includes some common utilities
+ * for all native code used in hadoop.
+ */
{code}
I don't think this comment is accurate.
{code}
+#include
{code}
Please move {{#include}}s to the relevant {{.c}} unless they're needed in the
{{.h}} directly. Doesn't look like it's needed here.
{code}
+memset(&addr, 0, sizeof(struct sockaddr_un));
{code}
I prefer to say {{memset(&x, 0, sizeof\(x\))}} so that the code is clearly
using the correct size. I don't feel too strongly about this though.
{code}
+addr.sun_family = AF_UNIX;
+if (bind(ud->sock, (struct sockaddr*)&addr, sizeof(sa_family_t)) < 0) {
{code}
This seems to be using the Linux-proprietary "abstract namespace". If we do
this it should be a Linux-specific name, not "unixDomainSock" which implies
that the code is portable to other UNIXes such as Darwin/Mac OS or Solaris or
FreeBSD.
The abstract socket API is documented at
http://www.kernel.org/doc/man-pages/online/pages/man7/unix.7.html
(If I'm wrong and the abstract sockets are supported by other OSes then great!
but I'm pretty sure they're not.)
Talking to Colin offline we confirmed that abstract sockets are Linux-specific,
but he pointed out that {{unixDomainSockCreateAndBind}} handles both abstract
sockets and named sockets (in the {{if(jpath)}} branch). So the name is OK but
we need a comment calling out the abstract socket use case. The Linux-specific
code will compile OK on other OSes, but it might be useful if the exception
message says "your OS requires an explicit path" on non-Linux (using an
{{#ifndef __linux__}} perhaps).
The control flow is a little confusing but not too bad, it could use a comment
perhaps something like {{/* Client requested abstract socket (see unix(7) for
details) by setting path = null. */}} in the abstract path case.
{code}
+ if (!jpath) {
... 20 lines of code
+ } else {
... 10 lines of code
+ }
{code}
I'd reorder them to {code}if (jpath) { ... } else { /* !jpath */ ... } {code}
as it's one less bit-flip to think about.
Could you explain the benefits of using abstract sockets? Why is it better
than a named socket? Ideally in a code comment near the implementation, or in
this Jira.
{code}
+ jthr = newNativeIOException(env, errno, "error autobinding "
+ "PF_UNIX socket: ");
{code}
I don't recognize the phrase "autobinding". Is that something specific to
abstract sockets? if so it can be described in the documentation.
{code}
+ jthr = newNativeIOException(env, EIO, "getsockname(): "
+ "autobound abstract socket identifier started with / ");
{code}
Most of your newnativeIOException texts end with {{: "}} but this one ends with
{{/ "}}. Best to be consistent.
{code}
+jthrowable unixDomainSetupSockaddr(JNIEnv *env, const char *id,
{code}
I think this function can be static, right?
{code}
+#define RETRY_ON_EINTR(ret, expr) do { \
+ ret = expr; \
+} while ((ret == -1) && (errno == EINTR));
{code}
This probably wants a maximum retry count (hardcoding 100 or thereabouts should
be fine).
{code}
+static ssize_t safe_write(int fd, const void *b, size_t c)
+{
+ int res;
+
+ while (c > 0) {
+res = write(fd, b, c);
+if (res < 0) {
+ if (errno != EINTR)
+return -errno;
+ continue;
+}
+c -= res;
+b = (char *)b + res;
{code}
* I'd use a local {{char *p = b}} rather than having the cast in the loop.
* if write returns a too large value (which "cannot happen" but bugs happen)
and c underflows, since it's unsigned the loop will spin forever (2^63 is
forever). I'd explicitly check {{if (res > c) return;}} before decrementing c.
* {{write(2)}} returns the number written. Seems like we should do that here
too. If the user calls safe_write(100), we write 50 and then get ENOSPC, we
should return 50 I think. But I'm not sure, maybe that's not the right
interface contract here.
{code}
+ if (!cmsg->cmsg_type == SCM_RIGHTS) {
{code}
Should be {{if (cmsg_type != SCM\_RIGHTS)}}.
{code}
+ if (setsockopt(sock, SOL_SOCKET, SO_RCVTIMEO, (char *)&timeou
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472927#comment-13472927
]
Hadoop QA commented on HADOOP-6311:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12548504/HADOOP-6311.021.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1595//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1595//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch,
> HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch,
> HADOOP-6311.020.patch, HADOOP-6311.021.patch, HADOOP-6311-0.patch,
> HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465799#comment-13465799
]
Hadoop QA commented on HADOOP-6311:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12547013/HADOOP-6311.020b.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1542//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1542//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch,
> HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020b.patch,
> HADOOP-6311.020.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch,
> hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465753#comment-13465753 ] Colin Patrick McCabe commented on HADOOP-6311: -- This one has no code derived from Android. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020.patch, > HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465363#comment-13465363 ] Arun C Murthy commented on HADOOP-6311: --- Colin - thanks for pushing on this, kinda fell off my radar. The original patch from Todd had code derived from Android, is that still the case? > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020.patch, > HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465302#comment-13465302
]
Hadoop QA commented on HADOOP-6311:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12546948/HADOOP-6311.020.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
12 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common:
org.apache.hadoop.ha.TestZKFailoverController
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1539//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/1539//console
This message is automatically generated.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Affects Versions: 0.20.0
>Reporter: Todd Lipcon
>Assignee: Colin Patrick McCabe
> Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch,
> HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311.020.patch,
> HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452622#comment-13452622 ] Hadoop QA commented on HADOOP-6311: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544546/HADOOP-6311.018.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 12 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1433//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1433//console This message is automatically generated. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311.016.patch, HADOOP-6311.018.patch, HADOOP-6311-0.patch, > HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452500#comment-13452500 ] Hadoop QA commented on HADOOP-6311: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544539/HADOOP-6311.017.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1430//console This message is automatically generated. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311.016.patch, HADOOP-6311.017.patch, HADOOP-6311-0.patch, > HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442740#comment-13442740 ] Hadoop QA commented on HADOOP-6311: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542648/HADOOP-6311.016.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 12 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1367//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1367//console This message is automatically generated. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311.016.patch, HADOOP-6311-0.patch, HADOOP-6311-1.patch, > hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13442312#comment-13442312 ] Hadoop QA commented on HADOOP-6311: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542580/HADOOP-6311.014.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 12 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.io.nativeio.TestNativeIO org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1363//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1363//console This message is automatically generated. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311.014.patch, > HADOOP-6311-0.patch, HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425141#comment-13425141 ] Brock Noland commented on HADOOP-6311: -- >From my perspective, go ahead! FWIW, the tests that failed in >TestLocalSockets passed on my RHEL 6.2 machine. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311-0.patch, > HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425130#comment-13425130 ] Colin Patrick McCabe commented on HADOOP-6311: -- Hi Brock / Todd, Are you guys still working on this? If your attention has been taken by other things, I'll post an updated version of this so we can get it in. C. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311-0.patch, > HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280689#comment-13280689 ] Hadoop QA commented on HADOOP-6311: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528428/HADOOP-6311-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.net.unix.TestLocalSockets org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1016//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1016//console This message is automatically generated. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311-0.patch, > HADOOP-6311-1.patch, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280140#comment-13280140 ] Brock Noland commented on HADOOP-6311: -- Agreed on all accounts. I will update the patch this morning and post this afternoon. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311-0.patch, > hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279843#comment-13279843 ] Todd Lipcon commented on HADOOP-6311: - Thanks for updating this, Brock! A few comments looking over the patch: - get_pw_buflen now seems to show up twice in NativeIO.c - we need to add interface annotations to the new public classes - I guess LimitedPrivate for HDFS is the right choice - in java_opt_to_real, I think we should use javah-produced constants to get the numeric values corresponding to java.net.SocketOptions here. The case statements with magic number ints are a little hard to explain If you dont have time to do the above, I can rev the patch > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, HADOOP-6311-0.patch, > hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078439#comment-13078439 ] Todd Lipcon commented on HADOOP-6311: - Thanks for reviewing - I agree some more cleanup and removal of code could be done. The efficiency shouldn't matter, as the intended use case here is really only to transfer open file descriptors and not to actually transfer bulk data. If people are on board with this general idea, I'll continue the cleanup, reformat the code to conform to our standards, and deal with the issues pointed out above. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078433#comment-13078433 ] Amr Awadallah commented on HADOOP-6311: --- I am out of office on vacation for next couple of days. I will be slower than usual in responding to emails. If this is urgent then please call my cell phone (or send an SMS), otherwise I will reply to your email when I get back. Thanks for your patience, -- amr > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078432#comment-13078432 ] Robert Joseph Evans commented on HADOOP-6311: - For what it is worth I thought it might be good to review the patch as a second opinion on its quality and maintainability. I did not really look at any of the Makefile/Configure changes as they are mostly auto generated. The following are the issues that I came up with. LocalSocket.java remove methods with UnsupportedOperationException unless we plan on implementing them. LocalSocketImpl.java may be very slow for large byte array reads and writes. I'm not sure how likely this is but if the array is large enough the java splits it into multiple locations then it might copy the data multiple times to comply with the JNI interface. It is not a correctness issues, just potentially a performance issue. LocalSocketImpl.java Potential race condition between reads/writes and closes (there is inconsistent locking with fd). LocalSocketImpl.java remove supportsUrgentData and sendUrgentData unless we plan on implementing them in the future, similar with getSockAddress() LocalSocketImpl.java in getOption I understand having a pattern in place to be able to return different types, but it is always int so can we get rid of the switch or combine the returns together. JNIHelp.cpp does not have an apache license in it. org_apache_hadoop_net_unix_LocalSocketImpl.cpp in make_sockaddr_un and java_opt_to_real I don't really like it writing to STDERR, it would be better to try and throw an exception instead. org_apache_hadoop_net_unix_LocalSocketImpl.cpp in Java_org_apache_hadoop_net_unix_LocalSocketImpl_connectLocal if an Exception occurred when calling jniGetFDFromFileDescriptor then nameUtf8 appears to be leaked. org_apache_hadoop_net_unix_LocalSocketImpl.cpp in Java_org_apache_hadoop_net_unix_LocalSocketImpl_bindLocal unlink is called regardless of namespace, and all errors with it are ignored. org_apache_hadoop_net_unix_LocalSocketImpl.cpp in JNI_OnLoad still has a reference to android in an commented out error log/ In the test please address the TODO you left in there about being a timeout exception instead of an IOException > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075987#comment-13075987 ] Todd Lipcon commented on HADOOP-6311: - To reply to a really old comment that I missed: bq. Todd, have you considered simply providing a jar and c-library from( or is derived from) the the Android code and making it available for hadoop? This way one can track the bug fixes and improvements in original Andriod. The issue is that Android is a giant monolithic project -- this code comes from its equivalent of the JDK runtime, so it doesn't have its own standalone .so file or jar. I could certainly take this patch and put it on my github (or cloudera's github) as a standalone Java library "junixsockets" or something. In the past, people have been skeptical of this kind of contribution and have preferred to have code pulled into Hadoop itself. Taking the code out of Android and adapting it to our needs is also beneficial in that we can re-use other utility code we already have in Hadoop common. For example, the patch in-progress for trunk reuses the NativeIOException that was developed during the security work. Regarding tracking bug fixes from Android, it's worth noting that in the nearly two years since this ticket was filed, there was only one very-minor bug fix in the Android code: http://android.git.kernel.org/?p=platform/frameworks/base.git;a=history;f=core/java/android/net/LocalSocket.java;h=3ee8a80e9968e0bf8c8c8b4d0ee3076476db4ec9;hb=HEAD So, I don't really foresee major maintenance issues in duplicating the code. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032120#comment-13032120 ] Jason Rutherglen commented on HADOOP-6311: -- Sorry, misspoke, the patch option wasn't given in Jira so there's no file, however the patch was mistakenly for Hadoop trunk. The port to 0.20-append is a bit more complex. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032096#comment-13032096 ] Hadoop QA commented on HADOOP-6311: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12476942/6311-trunk-inprogress.txt against trunk revision 1102093. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/437//console This message is automatically generated. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Affects Versions: 0.20.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13031997#comment-13031997 ] Jason Rutherglen commented on HADOOP-6311: -- bq. It may be some work to backport onto 20-append I think I'll try to backport it to 20-append in order to test and benchmark with the HBASE-3529. Unless HBase is going to upgrade to a newer Hadoop version soon? > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: 6311-trunk-inprogress.txt, hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022149#comment-13022149 ] Nichole Treadway commented on HADOOP-6311: -- Has anyone tested this on the Hadoop 0.20 Append branch? We are trying to build this but haven't been able to get this to successfully build. It always complains about some Zlib dependency issues. We can build Hadoop-0.20-Append without the patch, but it fails when patched with 6311. What are the dependencies for building this? > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765669#action_12765669 ] Sanjay Radia commented on HADOOP-6311: -- Todd, have you considered simply providing a jar and c-library from( or is derived from) the the Android code and making it available for hadoop? This way one can track the bug fixes and improvements in original Andriod. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765261#action_12765261 ] Konstantin Shvachko commented on HADOOP-6311: - > I will certainly look into Raghu's proposed improvement ... Thanks, this would be really good. Let's try what we can with internal solutions then turn to external ones. Is Android the best one? There should be alternatives there as well. Btw., does this library work on Windows, Solaris, etc.? I think Dhruba is making a good point in HDFS-347 that until we really understand what causes the slowdown we cannot efficiently fight it. On a side note, this project becomes fairly complicated, should we consider branching it? > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765230#action_12765230 ] Owen O'Malley commented on HADOOP-6311: --- I'll remove my -1, but stick by a -0. I think this is a very bad idea. Forking code out of existing projects is a very dangerous practice and this is a huge chunk of code to be incorporating into Hadoop. This will become critical code in the heart of HDFS that none of us developed. That is troubling. I agree that we should not commit this until HDFS-347 is ready to commit. I suspect that putting it into HDFS is harder than it is worth, since HDFS doesn't have a native component already. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[
https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765199#action_12765199
]
Todd Lipcon commented on HADOOP-6311:
-
bq. You propose to commit MBs of new code, which we don't know nothing about
and take the burden of supporting it, if I understood Doug correctly.
As I said above, the patch is huge due to having to rerun autoconf/aclocal/etc.
The actual amount of new code in the patch:
{noformat}
build.xml | 10 +
.../org/apache/hadoop/net/unix/Credentials.java| 52 +
.../apache/hadoop/net/unix/LocalServerSocket.java | 122 +
.../org/apache/hadoop/net/unix/LocalSocket.java| 288 +
.../apache/hadoop/net/unix/LocalSocketAddress.java | 95 +
.../apache/hadoop/net/unix/LocalSocketImpl.java| 499 +
.../src/org/apache/hadoop/net/unix/JNIHelp.cpp | 85 +
.../src/org/apache/hadoop/net/unix/JNIHelp.h | 76 +
.../src/org/apache/hadoop/net/unix/Makefile.am | 51 +
.../hadoop/net/unix/java_io_FileDescriptor.cpp | 201 +
.../org_apache_hadoop_net_unix_LocalSocketImpl.cpp | 1012 +
.../apache/hadoop/net/unix/TestLocalSockets.java | 208 +
{noformat}
So, this isn't a tiny patch, but it's certainly not huge either. If you
discount comments and whitespace, it's 867 lines of cpp, 628 lines of java (120
are test).
If you prefer, I can rip out all of the code but the very minimal required, but
I don't consider this a particularly large patch.
As for alternatives, I think many of them were thrown out due to security
concerns in HDFS-347. I will certainly look into Raghu's proposed improvement
to FSInputChecker, and if that gets the same gains as this, I'm happy to throw
out all of the localsockets complication.
> Add support for unix domain sockets to JNI libs
> ---
>
> Key: HADOOP-6311
> URL: https://issues.apache.org/jira/browse/HADOOP-6311
> Project: Hadoop Common
> Issue Type: New Feature
> Components: native
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Attachments: hadoop-6311.txt
>
>
> For HDFS-347 we need to use unix domain sockets. This JIRA is to include a
> library in common which adds a o.a.h.net.unix package based on the code from
> Android (apache 2 license)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765195#action_12765195 ] Konstantin Shvachko commented on HADOOP-6311: - I am really hesitant we should commit to Hadoop sources from another project. As Owen mentioned the main concern is the support of the code. You propose to commit MBs of new code, which we don't know nothing about and take the burden of supporting it, if I understood Doug correctly. - Ok, so does the purpose justifies the sacrifice? Not sure that even 30% improvement in read performance justifies this. Looking at your benchmarks I think the number will go down to 10-20% for MapReduce in general because of data-node overhead, and since not all task reads are local. - Are there alternatives? You probably need at least to consider alternatives, and explain why they don't work. Seems like Dhruba and Raghu made attempts to turn this to solving problems inside data-node code, which has good chances imo to reach the same results without massive infusion of alien sources. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765171#action_12765171 ] Todd Lipcon commented on HADOOP-6311: - bq. Okay, but I'd then vote that we not commit this until either there's another dependency on it that's committed or we're about to commit HDFS-347. +1 on not committing until we're ready to do the whole thing. However, JIRA won't let me turn it into a subtask of HDFS-347: "Only non-sub-task issues from the same project (HADOOP) can be selected." > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765169#action_12765169 ] Doug Cutting commented on HADOOP-6311: -- In other words, I view this as a sub-task of HDFS-347. If you agree, perhaps you could convert it? > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765168#action_12765168 ] Doug Cutting commented on HADOOP-6311: -- > I put this code in common since that's the easiest integration point for > additional native code. Okay, but I'd then vote that we not commit this until either there's another dependency on it that's committed or we're about to commit HDFS-347. Longer term, perhaps we'll add native code to the HDFS project and HDFS-347 could become a self-contained HDFS-only patch. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765161#action_12765161 ] Todd Lipcon commented on HADOOP-6311: - bq. This might be acceptable if you can get the Android project to make releases that we can use. (Or if they split it out into a stand alone project that they also use...) I think there's little chance of this. The original code has (a) some android specific things like a special android socket namespace, (b) reliance on their custom build system, and (c) reliance on a bunch of other utility code elsewhere in Android. This is also such a small part of Android that I am confident they'd have no interest in splitting it out to a separate library - it would just be a waste of time for them. Regarding maintenance and bugs, I think if you look at the code you'll see that it's not super complicated; obviously there's always a chance for bugs, especially on different platforms, but I don't see this code evolving much from where it is today (in Android it has had only one edit in the year since it was contributed). Also, since Android targets a very specific platform (and specific nonstandard JVM, even) I don't think we'd see the same bugs they would in practice. bq. until we have another use of unix domain sockets within Hadoop, or at least have another expected use, keep the code with its single user in HDFS. I put this code in common since that's the easiest integration point for additional native code. If I were to put it in an entirely separate JNI library, we'd have to make changes to (or duplicate) NativeCodeLoader, ask users to install multiple native libraries, add appropriate build infrastructure to the hdfs build, etc. I think it makes most sense to have a single "native-code based optimizations" library that contains these optional extensions in common even if there is currently only a single user. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765154#action_12765154 ] Doug Cutting commented on HADOOP-6311: -- Forking is certainly often not an optimal path. That said, forking is quite different than using a snapshot build. In that case we committed the jar. Here, Todd's proposing we commit the source, properly forking. We'd be taking on the maintenance of this code, simply using the Android code as a starting point. That might be warranted if Android is not a project that's in the business of producing independent utility jar file releases, which it does not appear to be. If you agree with the above analysis, then the question has little to do with the source of the code but rather with whether Hadoop's codebase should include unix domain socket utilities. My general preference is to refactor later rather than earlier: until we have another use of unix domain sockets within Hadoop, or at least have another expected use, keep the code with its single user in HDFS. Do you have another expected use? > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6311) Add support for unix domain sockets to JNI libs
[ https://issues.apache.org/jira/browse/HADOOP-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765128#action_12765128 ] Owen O'Malley commented on HADOOP-6311: --- -1 Forking part of Android into Hadoop, while legal, is a really bad idea. *We* will become responsible for maintaining the code and will need to track all of the bug fixes that go into it. This is a huge maintenance burden. This might be acceptable if you can get the Android project to make releases that we can use. (Or if they split it out into a stand alone project that they also use...) For some related history, look at the pain that happened when we used a snapshot build from Apache Commons CLI. We eventually had to roll back to the released version including downgrading all of our usage of the new features. > Add support for unix domain sockets to JNI libs > --- > > Key: HADOOP-6311 > URL: https://issues.apache.org/jira/browse/HADOOP-6311 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hadoop-6311.txt > > > For HDFS-347 we need to use unix domain sockets. This JIRA is to include a > library in common which adds a o.a.h.net.unix package based on the code from > Android (apache 2 license) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
