[
https://issues.apache.org/jira/browse/HDFS-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HDFS-14325:
--------------------------------
Description:
The usage of [errno|http://man7.org/linux/man-pages/man3/errno.3.html] in
libhdfs has gone through several changes in the past: HDFS-3675, HDFS-4997,
HDFS-3579, HDFS-8407, etc.
As a result of these changes, some libhdfs functions set {{errno}} to 0 on
success ({{hadoopReadZero}}, {{hdfsListDirectory}}), while several set
{{errno}} to a meaningful value only on error.
libhdfs++ on the other hand sets {{errno}} to 0 for all (successful) libhdfs++
operations. See
[this|https://issues.apache.org/jira/browse/HDFS-10511?focusedCommentId=15322696&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15322696]
comment in HDFS-10511 for why that was done.
The inconsistent behavior between libhdfs and libhdfs++ causes issues for tests
such as {{test_hdfs_ext_hdfspp_test_shim_static}} which uses a shim layer
({{tests/hdfs_shim.c}}) that delegates to both {{hdfs.c}} and {{hdfs.cc}} for
various operations (e.g. opening / closing files uses both APIs, {{hdfsWrite}}
delegates to libhdfs since libhdfs++ does not support writes yet). The tests
expect {{errno}} to be set to 0 after successful operations against the shim
layer. Since libhdfs is not guaranteed to set {{errno}} to 0 on success, tests
can start failing.
One example of the inconsistency causing issues is HDFS-14111, the patch for
HDFS-14111 happens to change the {{errno}} from 0 to 2 for {{hdfsCloseFile}}.
However, from libhdfs's perspective this seems to be by design. Quoting from
the {{errno}} C docs:
{quote}The value in errno is significant only when the return value of the call
indicated an error (i.e., -1 from most system calls; -1 or NULL from most
library functions); a function that succeeds is allowed to change errno.
{quote}
I was not able to pin down why the patch for HDFS-14111 changed the {{errno}}
value, but I isolated the change to the {{FileSystem#close}} call. Most likely,
some C function invoked as a result of calling {{#close}} failed and changed
the {{errno}} value, but the {{#close}} was still able to succeed (this is most
likely expected behavior, see
[this|https://issues.apache.org/jira/browse/HDFS-8407?focusedCommentId=14551225&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14551225]
comment in HDFS-8407 for further validation).
Going forward we could (1) set {{errno}} to 0 for all successful libhdfs
functions, which would make the libhdfs behavior consistent with the libhdfs++
behavior, or (2) we could live with the discrepancy, which would require
modifying from the libhdfs++ shim tests (which assert that {{errno}} is 0 after
certain operations) and just document the difference in behavior.
was:
The usage of [errno|http://man7.org/linux/man-pages/man3/errno.3.html] in
libhdfs has gone through several changes in the past: HDFS-3675, HDFS-4997,
HDFS-3579, HDFS-8407, etc.
As a result of these changes, some libhdfs functions set {{errno}} to 0 on
success ({{hadoopReadZero}}, {{hdfsListDirectory}}), while several set
{{errno}} to a meaningful value only on error.
libhdfs++ on the other hand sets {{errno}} to 0 for all (successful) libhdfs++
operations. See this comment in HDFS-10511 for why that was done.
The inconsistent behavior between libhdfs and libhdfs++ causes issues for tests
such as {{test_hdfs_ext_hdfspp_test_shim_static}} which uses a shim layer
({{tests/hdfs_shim.c}}) that delegates to both {{hdfs.c}} and {{hdfs.cc}} for
various operations (e.g. opening / closing files uses both APIs, {{hdfsWrite}}
delegates to libhdfs since libhdfs++ does not support writes yet). The tests
expect {{errno}} to be set to 0 after successful operations against the shim
layer. Since libhdfs is not guaranteed to set {{errno}} to 0 on success, tests
can start failing.
One example of the inconsistency causing issues is HDFS-14111, the patch for
HDFS-14111 happens to change the {{errno}} from 0 to 2 for {{hdfsCloseFile}}.
However, from libhdfs's perspective this seems to be by design. Quoting from
the {{errno}} C docs:
{quote}The value in errno is significant only when the return value of the call
indicated an error (i.e., -1 from most system calls; -1 or NULL from most
library functions); a function that succeeds is allowed to change errno.
{quote}
I was not able to pin down why the patch for HDFS-14111 changed the {{errno}}
value, but I isolated the change to the {{FileSystem#close}} call. Most likely,
some C function invoked as a result of calling {{#close}} failed and changed
the {{errno}} value, but the {{#close}} was still able to succeed (this is most
likely expected behavior, see this comment in HDFS-8407 for further validation).
Going forward we could (1) set {{errno}} to 0 for all successful libhdfs
functions, which would make the libhdfs behavior consistent with the libhdfs++
behavior, or (2) we could live with the discrepancy, which would require
modifying from the libhdfs++ shim tests (which assert that {{errno}} is 0 after
certain operations) and just document the difference in behavior.
> Revise usage of errno in libhdfs
> --------------------------------
>
> Key: HDFS-14325
> URL: https://issues.apache.org/jira/browse/HDFS-14325
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> The usage of [errno|http://man7.org/linux/man-pages/man3/errno.3.html] in
> libhdfs has gone through several changes in the past: HDFS-3675, HDFS-4997,
> HDFS-3579, HDFS-8407, etc.
> As a result of these changes, some libhdfs functions set {{errno}} to 0 on
> success ({{hadoopReadZero}}, {{hdfsListDirectory}}), while several set
> {{errno}} to a meaningful value only on error.
> libhdfs++ on the other hand sets {{errno}} to 0 for all (successful)
> libhdfs++ operations. See
> [this|https://issues.apache.org/jira/browse/HDFS-10511?focusedCommentId=15322696&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15322696]
> comment in HDFS-10511 for why that was done.
> The inconsistent behavior between libhdfs and libhdfs++ causes issues for
> tests such as {{test_hdfs_ext_hdfspp_test_shim_static}} which uses a shim
> layer ({{tests/hdfs_shim.c}}) that delegates to both {{hdfs.c}} and
> {{hdfs.cc}} for various operations (e.g. opening / closing files uses both
> APIs, {{hdfsWrite}} delegates to libhdfs since libhdfs++ does not support
> writes yet). The tests expect {{errno}} to be set to 0 after successful
> operations against the shim layer. Since libhdfs is not guaranteed to set
> {{errno}} to 0 on success, tests can start failing.
> One example of the inconsistency causing issues is HDFS-14111, the patch for
> HDFS-14111 happens to change the {{errno}} from 0 to 2 for {{hdfsCloseFile}}.
> However, from libhdfs's perspective this seems to be by design. Quoting from
> the {{errno}} C docs:
> {quote}The value in errno is significant only when the return value of the
> call indicated an error (i.e., -1 from most system calls; -1 or NULL from
> most library functions); a function that succeeds is allowed to change errno.
> {quote}
> I was not able to pin down why the patch for HDFS-14111 changed the {{errno}}
> value, but I isolated the change to the {{FileSystem#close}} call. Most
> likely, some C function invoked as a result of calling {{#close}} failed and
> changed the {{errno}} value, but the {{#close}} was still able to succeed
> (this is most likely expected behavior, see
> [this|https://issues.apache.org/jira/browse/HDFS-8407?focusedCommentId=14551225&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14551225]
> comment in HDFS-8407 for further validation).
> Going forward we could (1) set {{errno}} to 0 for all successful libhdfs
> functions, which would make the libhdfs behavior consistent with the
> libhdfs++ behavior, or (2) we could live with the discrepancy, which would
> require modifying from the libhdfs++ shim tests (which assert that {{errno}}
> is 0 after certain operations) and just document the difference in behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
