[jira] [Work started] (HDFS-14342) WebHDFS: expose NEW_BLOCK flag in APPEND operation
[ https://issues.apache.org/jira/browse/HDFS-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14342 started by Anatoli Shein. > WebHDFS: expose NEW_BLOCK flag in APPEND operation > -- > > Key: HDFS-14342 > URL: https://issues.apache.org/jira/browse/HDFS-14342 > Project: Hadoop HDFS > Issue Type: Task > Components: webhdfs >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > > After the support for variable length blocks was added (HDFS-3689), we should > expose the NEW_BLOCK flag of APPEND operation in webhdfs, so that this > functionality will be usable over the rest api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14342) WebHDFS: expose NEW_BLOCK flag in APPEND operation
[ https://issues.apache.org/jira/browse/HDFS-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14342: - Attachment: HDFS-14342.000.patch Status: Patch Available (was: In Progress) Attaching a quick proof of concept patch that demos this functionality. > WebHDFS: expose NEW_BLOCK flag in APPEND operation > -- > > Key: HDFS-14342 > URL: https://issues.apache.org/jira/browse/HDFS-14342 > Project: Hadoop HDFS > Issue Type: Task > Components: webhdfs >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14342.000.patch > > > After the support for variable length blocks was added (HDFS-3689), we should > expose the NEW_BLOCK flag of APPEND operation in webhdfs, so that this > functionality will be usable over the rest api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14342) WebHDFS: expose NEW_BLOCK flag in APPEND operation
Anatoli Shein created HDFS-14342: Summary: WebHDFS: expose NEW_BLOCK flag in APPEND operation Key: HDFS-14342 URL: https://issues.apache.org/jira/browse/HDFS-14342 Project: Hadoop HDFS Issue Type: Task Components: webhdfs Reporter: Anatoli Shein Assignee: Anatoli Shein After the support for variable length blocks was added (HDFS-3689), we should expose the NEW_BLOCK flag of APPEND operation in webhdfs, so that this functionality will be usable over the rest api. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685398#comment-16685398 ] Anatoli Shein commented on HDFS-11520: -- Patch 022 - lower num_iterations to 500 to make sure we don't run out of sockets. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.020.patch, HDFS-11520.021.patch, HDFS-11520.022.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.022.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.020.patch, HDFS-11520.021.patch, HDFS-11520.022.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685198#comment-16685198 ] Anatoli Shein commented on HDFS-11520: -- New patch (021) includes more comments on how things work. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.020.patch, HDFS-11520.021.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.021.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.020.patch, HDFS-11520.021.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673619#comment-16673619 ] Anatoli Shein commented on HDFS-14047: -- Okay, I finally got to the bottom of this issue and fixed it in the new patch (001). This problem apparently started happening after HADOOP-15495 (Upgrade common-lang version to 3.7 in hadoop-common-project and hadoop-tools) since the reference to: org.apache.commons.lang.exception.ExceptionUtils was missed and not updated to: org.apache.commons.lang3.exception.ExceptionUtils in exception.c [~James C], could you please help reviewing this? > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch, HDFS-14047.001.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Attachment: HDFS-14047.001.patch > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch, HDFS-14047.001.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673286#comment-16673286 ] Anatoli Shein commented on HDFS-14047: -- So the CI test passed with these lines commented out. After looking at this more, it looks like the issue is with Libhdfs. Libhdfs++ hits this as well just because it falls back to Libhdfs for these operations, so fixing this issue might be more tricky. Should we remove the two failing lines for the time being until this gets fixed? > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Description: Currently the native client CI tests break deterministically with these errors: Libhdfs 1 - test_test_libhdfs_threaded_hdfs_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) Libhdfs++ 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) was: Currently the native client CI tests break deterministically with these errors: Libhdfs 1 - test_test_libhdfs_threaded_hdfs_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) Libhdfs++ 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Attachment: HDFS-14047.000.patch > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14047: - Status: Patch Available (was: In Progress) Patch 000 - comment out hdfsGetLastExceptionRootCause first to make sure CI tests pass this way. > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14047 started by Anatoli Shein. > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14022) Failing CTEST test_libhdfs
[
https://issues.apache.org/jira/browse/HDFS-14022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673095#comment-16673095
]
Anatoli Shein commented on HDFS-14022:
--
Looks like the remaining failures on trunk are caused by
hdfsGetLastExceptionRootCause function call in test_libhdfs_threaded.c. I just
created HDFS-14047 to look into this.
> Failing CTEST test_libhdfs
> --
>
> Key: HDFS-14022
> URL: https://issues.apache.org/jira/browse/HDFS-14022
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Major
>
> Here are list of recurring failures that are seen, there seems to be a
> problem with
> invoking the build() in MiniDFSClusterBuilder, there are several failures ( 2
> cores related to it), in the function
> struct NativeMiniDfsCluster* nmdCreate(struct NativeMiniDfsConf *conf)
> {
>jthr = invokeMethod(env, , INSTANCE, bld, MINIDFS_CLUSTER_BUILDER,
> "build", "()L" MINIDFS_CLUSTER ";"); --->
> }
> Failed CTEST tests
> test_test_libhdfs_threaded_hdfs_static
> test_test_libhdfs_zerocopy_hdfs_static
> test_libhdfs_threaded_hdfspp_test_shim_static
> test_hdfspp_mini_dfs_smoke_hdfspp_test_shim_static
> libhdfs_mini_stress_valgrind_hdfspp_test_static
> memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static
> test_libhdfs_mini_stress_hdfspp_test_shim_static
> test_hdfs_ext_hdfspp_test_shim_static
>
> Details of the failures:
> a) test_test_libhdfs_threaded_hdfs_static
> hdfsOpenFile(/tlhData0001/file1):
> FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
> error:
> (unable to get root cause for java.io.FileNotFoundException) --->
> (unable to get stack trace for java.io.FileNotFoundException)
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180
> with NULL return return value (errno: 2): expected substring: File does not
> exist
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336
> with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti,
> fs, )
> hdfsOpenFile(/tlhData/file1):
> FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
> error:
> (unable to get root cause for java.io.FileNotFoundException)
> b) test_test_libhdfs_zerocopy_hdfs_static
> nmdCreate: Builder#build error:
> (unable to get root cause for java.lang.RuntimeException)
> (unable to get stack trace for java.lang.RuntimeException)
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_zerocopy.c:253
> (errno: 2): got NULL from cl
> Failure:
> struct NativeMiniDfsCluster* nmdCreate(struct NativeMiniDfsConf *conf)
> jthr = invokeMethod(env, , INSTANCE, bld, MINIDFS_CLUSTER_BUILDER,
> "build", "()L" MINIDFS_CLUSTER ";"); ===> Failure
> if (jthr) {
> printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "nmdCreate: Builder#build");
> goto error;
> }
> }
> c) test_libhdfs_threaded_hdfspp_test_shim_static
> hdfsOpenFile(/tlhData0002/file1):
> FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;)
> error:
> (unable to get root cause for java.io.FileNotFoundException) --->
> (unable to get stack trace for java.io.FileNotFoundException)
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180
> with NULL return return value (errno: 2): expected substring: File does not
> exist
> TEST_ERROR: failed on
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336
> with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti,
> fs, )
> d)
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x0078c513, pid=16765, tid=0x7fc4449717c0
> #
> # JRE version: OpenJDK Runtime Environment (8.0_181-b13) (build
> 1.8.0_181-8u181-b13-0ubuntu0.16.04.1-b13)
> # Java VM: OpenJDK 64-Bit Server VM (25.181-b13 mixed mode linux-amd64
> compressed oops)
> # Problematic frame:
> # C [hdfs_ext_hdfspp_test_shim_static+0x38c513]
> #
> # Core dump written. Default location:
> /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/main/native/libhdfspp/tests/core
> or core.16765
> #
> # An error report file with more information is saved as:
> #
>
[jira] [Created] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
Anatoli Shein created HDFS-14047: Summary: [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c Key: HDFS-14047 URL: https://issues.apache.org/jira/browse/HDFS-14047 Project: Hadoop HDFS Issue Type: Sub-task Components: libhdfs, native Reporter: Anatoli Shein Assignee: Anatoli Shein Currently the native client CI tests break deterministically with these errors: Libhdfs 1 - test_test_libhdfs_threaded_hdfs_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) Libhdfs++ 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 with NULL return return value (errno: 2): expected substring: File does not exist [exec] TEST_ERROR: failed on /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, fs, ) [exec] hdfsOpenFile(/tlhData0001/file1): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: [exec] (unable to get root cause for java.io.FileNotFoundException) [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667567#comment-16667567 ] Anatoli Shein commented on HDFS-11520: -- Patch 020 - now trying to get a passing CI run by commenting out a hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.020.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.020.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.020.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667546#comment-16667546 ] Anatoli Shein commented on HDFS-14033: -- [~sunilg] As [~James C] correctly pointed out, these failing tests are pre-existing and not related to this patch, and we are planning to fix them in the near future as well. > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14033.000.patch, HDFS-14033.001.patch > > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14033: - Attachment: HDFS-14033.001.patch > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14033.000.patch, HDFS-14033.001.patch > > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667375#comment-16667375 ] Anatoli Shein commented on HDFS-14033: -- Thank for the quick review [~James C] In the new patch (001) I added version numbers of compilers that support thread_local for convenience. > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14033.000.patch, HDFS-14033.001.patch > > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14033: - Attachment: HDFS-14033.000.patch > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14033.000.patch > > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-14033: - Status: Patch Available (was: In Progress) Added a fix in hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14033.000.patch > > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14033 started by Anatoli Shein. > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667175#comment-16667175 ] Anatoli Shein edited comment on HDFS-12026 at 10/29/18 1:09 PM: Thanks for your feedback [~sunilg], [~vagarychen], and [~shv]. [~shv] in regard to your questions: 1. It shouldn't affect the runtime, and things should work without errors. 2. This is a good observation. We should probably disable the Libhdfs++ build based on thread_local support (which is the dependency missing from older compilers and needed for Libhdfs++ ). This way all the older systems (including old Centos and Ubuntu versions) will still build. We will just emit a warning saying Libhdfs++ library wasn't built. I just opened a jira for this: HDFS-14033 and should have a patch soon. was (Author: anatoli.shein): Thanks for your feedback [~sunilg], [~vagarychen], and [~shv]. [~shv] in regard to your questions: 1. It shouldn't affect the runtime, and things should work without errors. 2. This is a good observation. We should probably disable the Libhdfs++ build based on thread_local support (which is the dependency missing from older compilers and needed for Libhdfs++). This way all the older systems (including old Centos and Ubuntu versions) will still build. We will just emit a warning saying Libhdfs++ library wasn't built. I just opened a jira for this: HDFS-14033 and should have a patch soon. > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Blocker > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch, > HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch, > HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch, > HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flag: > -std=c++11 > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
Anatoli Shein created HDFS-14033: Summary: [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local Key: HDFS-14033 URL: https://issues.apache.org/jira/browse/HDFS-14033 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Anatoli Shein In order to still be able to build Hadoop on older systems (such as rhel 6) we need to disable libhdfs++ build on systems that do not support thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein reassigned HDFS-14033: Assignee: Anatoli Shein > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667175#comment-16667175 ] Anatoli Shein edited comment on HDFS-12026 at 10/29/18 1:07 PM: Thanks for your feedback [~sunilg], [~vagarychen], and [~shv]. [~shv] in regard to your questions: 1. It shouldn't affect the runtime, and things should work without errors. 2. This is a good observation. We should probably disable the Libhdfs++ build based on thread_local support (which is the dependency missing from older compilers and needed for Libhdfs++). This way all the older systems (including old Centos and Ubuntu versions) will still build. We will just emit a warning saying Libhdfs++ library wasn't built. I just opened a jira for this: HDFS-14033 and should have a patch soon. was (Author: anatoli.shein): Thanks for your feedback [~sunilg], [~vagarychen], and [~shv]. [~shv] in regard to your questions: 1. It shouldn't affect the runtime, and things should work without errors. 2. This is a good observation. We should probably disable the Libhdfs++ build based on thread_local support (which is the dependency missing from older compilers and needed for Libhdfs++). This way all the older systems (including old Centos and Ubuntu versions) will still build. We will just emit a warning saying Libhdfs++ library wasn't built. I just opened a jira for this: HDFS-14033 and should have a patch soon. > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Blocker > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch, > HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch, > HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch, > HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flag: > -std=c++11 > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667175#comment-16667175 ] Anatoli Shein commented on HDFS-12026: -- Thanks for your feedback [~sunilg], [~vagarychen], and [~shv]. [~shv] in regard to your questions: 1. It shouldn't affect the runtime, and things should work without errors. 2. This is a good observation. We should probably disable the Libhdfs++ build based on thread_local support (which is the dependency missing from older compilers and needed for Libhdfs++). This way all the older systems (including old Centos and Ubuntu versions) will still build. We will just emit a warning saying Libhdfs++ library wasn't built. I just opened a jira for this: HDFS-14033 and should have a patch soon. > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Blocker > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch, > HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch, > HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch, > HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flag: > -std=c++11 > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665383#comment-16665383 ] Anatoli Shein commented on HDFS-12026: -- Hi [~sunilg], So not building Libhdfs++ on RHEL6 shouldn't have any impact besides not being able to use Libhdfs++ to connect to hdfs. Also I don't think there could be any upgrade issues later as well. > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Blocker > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch, > HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch, > HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch, > HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flag: > -std=c++11 > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10513) libhdfs++: Ensure we're connected when FS methods are called
[ https://issues.apache.org/jira/browse/HDFS-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665276#comment-16665276 ] Anatoli Shein commented on HDFS-10513: -- This is just for a sanity check to make sure FileSystem::Connect has been called before FS methods are called. IIRC when I tested this previously, the error message that I was getting in this case was ambiguous. > libhdfs++: Ensure we're connected when FS methods are called > > > Key: HDFS-10513 > URL: https://issues.apache.org/jira/browse/HDFS-10513 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Priority: Major > > We can either transparently connect or complain if we're not -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665243#comment-16665243 ] Anatoli Shein commented on HDFS-12026: -- Thanks [~sunilg] and [~vagarychen] We can fix this quickly by just conditionally disabling libhdfs++ build in CMakeLists.txt if the OS is RHEL6. Libhdfs++ would still build on other platforms. Would that be okay? > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Blocker > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch, > HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch, > HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch, > HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flag: > -std=c++11 > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663993#comment-16663993 ] Anatoli Shein commented on HDFS-12026: -- [~James C] and I heavily rely on this functionality and would like to keep it. [~shv] Which versions of gcc and glibc are required to be supported? > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Blocker > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch, > HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch, > HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch, > HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flag: > -std=c++11 > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653923#comment-16653923 ] Anatoli Shein edited comment on HDFS-11520 at 10/17/18 5:32 PM: Patch 19: Testing trunk to make sure it is failing the CI run. was (Author: anatoli.shein): Path 19: Testing trunk to make sure it is failing the CI run. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653923#comment-16653923 ] Anatoli Shein commented on HDFS-11520: -- Path 19: Testing trunk to make sure it is failing the CI run. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.019.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653536#comment-16653536 ] Anatoli Shein commented on HDFS-11520: -- Looks like multiple tests are failing due to bad LD_LIBRARY_PATH. Patch 18: Fix for LD_LIBRARY_PATH to get a passing CI run. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.018.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.017.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652399#comment-16652399 ] Anatoli Shein commented on HDFS-11520: -- Temporarily commented out one unrelated bug to see if I can get a passing CI run. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.016.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651718#comment-16651718 ] Anatoli Shein commented on HDFS-11520: -- The new patch fixes the valgrind memory leak. Now ready for review. There are some (unrelated) unit test failures, I will look into fixing them next. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.015.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650787#comment-16650787 ] Anatoli Shein commented on HDFS-11520: -- The new patch fixes all previous race conditions, and adds stress testing by doing cancellations in a loop. Yet, there is a memory leak that shows up when running under valgrind, I will look into this next. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.014.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645292#comment-16645292 ] Anatoli Shein commented on HDFS-11520: -- In the new patch I broke the monolithic test into 5 tests, and added functionality to run them in a loop. Unfortunately I occasionally get segfaults still. Will look more into it next. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.013.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638333#comment-16638333 ] Anatoli Shein commented on HDFS-11520: -- Just added a patch where I added a test for failover (by hacking configs) and reduced code duplication by adding functions test_cancel_during_event(), test_cancel_after_retry_during_event(), and test_cancel_after_failover_during_event(). Ready for review. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.012.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628726#comment-16628726 ] Anatoli Shein commented on HDFS-11520: -- New patch has a fix for a race with retry case. Failover case works now according to manual testing, however still trying to figure out how to add a test. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.011.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.010.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16615289#comment-16615289 ] Anatoli Shein commented on HDFS-11520: -- In the new patch I added cancellation after retry. Next working on a failover case. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604420#comment-16604420 ] Anatoli Shein commented on HDFS-11520: -- In the new patch I fixed a race in the previous tests, and added a new test for the recursive calls. Next working on retry/failover. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.009.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590790#comment-16590790 ] Anatoli Shein commented on HDFS-11520: -- New patch adds cancellation during FlushPendingRequests, which does not swap the callback, and added a test. Recursive stuff seems to be working now as well, will add a test for that next, and will look at retry/failover. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.008.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.007.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: (was: HDFS-11520.006.patch) > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589297#comment-16589297 ] Anatoli Shein commented on HDFS-11520: -- Thanks for the review [~James C], In the new (checkpoint) patch I fixed the outdated/confusing comments, added another place for cancellation (during OnSendCompleted), added functionality for adding multiple requests to the same cancel handle (for recursive stuff). Still working on adding more tests and support for cancellation during recursion/retry/failover. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.006.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.006.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.006.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13429) libhdfs++ Expose a C++ logging API
[ https://issues.apache.org/jira/browse/HDFS-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589004#comment-16589004 ] Anatoli Shein commented on HDFS-13429: -- Thanks for the patch, [~James C], separating/making consistent C and C++ logging APIs is definitely important and needed. Unit tests look good as well. +1. > libhdfs++ Expose a C++ logging API > -- > > Key: HDFS-13429 > URL: https://issues.apache.org/jira/browse/HDFS-13429 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13429.000.patch, HDFS-13429.001.patch > > > The libhdfs++ C API supports taking function pointers for log plugins and > defines levels for components and severity. It'd be nice to have an > idiomatic C++ logging interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13745) libhdfs++: Fix race in FileSystem destructor
[ https://issues.apache.org/jira/browse/HDFS-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588924#comment-16588924 ] Anatoli Shein commented on HDFS-13745: -- Also, can we submit another CI run for this? Looks like the previous one didn't run for some reason. > libhdfs++: Fix race in FileSystem destructor > > > Key: HDFS-13745 > URL: https://issues.apache.org/jira/browse/HDFS-13745 > Project: Hadoop HDFS > Issue Type: Task > Components: native >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13745.000.patch > > > Whatever happens to have the last shared_ptr to the IoService will run > ~IoService when the shared_ptr goes out of scope. IoService's destructor is > responsible for joining all worker threads in the pool. Most callbacks now > own weak_ptr that can be promoted to a shared_ptr in order to post > new async tasks. If a callback object is the last thing holding the > IoService shared_ptr it's going to try to join the thread pool inside of one > of the thread pool's threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13745) libhdfs++: Fix race in FileSystem destructor
[ https://issues.apache.org/jira/browse/HDFS-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588921#comment-16588921 ] Anatoli Shein commented on HDFS-13745: -- Thanks for the patch [~James C]! I have just a few questions about this: 1. Is there a possibility that some task executed by the IoService will run forever? Should we add some timeout in BlockingStop method if we have been waiting too long? 2. In the hdfs_ioservice_test in longRunningCallback we sleep for just 1 second, which might not be enough since if there is some sort of system delay longer than 1 second the test might fail. Even though with any amount of sleep there is a chance of this happening, it might make sense to increase it to 2-3 seconds. > libhdfs++: Fix race in FileSystem destructor > > > Key: HDFS-13745 > URL: https://issues.apache.org/jira/browse/HDFS-13745 > Project: Hadoop HDFS > Issue Type: Task > Components: native >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13745.000.patch > > > Whatever happens to have the last shared_ptr to the IoService will run > ~IoService when the shared_ptr goes out of scope. IoService's destructor is > responsible for joining all worker threads in the pool. Most callbacks now > own weak_ptr that can be promoted to a shared_ptr in order to post > new async tasks. If a callback object is the last thing holding the > IoService shared_ptr it's going to try to join the thread pool inside of one > of the thread pool's threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514023#comment-16514023 ] Anatoli Shein commented on HDFS-11520: -- Okay, so it looks like shadedclient fails now on the same patch that was passing just 2 days ago. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513868#comment-16513868 ] Anatoli Shein edited comment on HDFS-11520 at 6/15/18 2:17 PM: --- Poking the CI system to see if previous patch (002) still builds fine. (resubmitted it as 005). was (Author: anatoli.shein): Poking the CI system to see if previous patch (002) still builds fine. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.005.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513868#comment-16513868 ] Anatoli Shein commented on HDFS-11520: -- Poking the CI system to see if previous patch (002) still builds fine. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16513686#comment-16513686 ] Anatoli Shein commented on HDFS-11520: -- Not sure why shadedclient has errors (cannot find output). Trying again without the variadic stuff in the new patch. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.004.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512965#comment-16512965 ] Anatoli Shein commented on HDFS-11520: -- In the new patch I added tests for canceling all RPCs. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: (was: HDFS-11520.002.patch) > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.003.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.002.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.002.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.002.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16510002#comment-16510002 ] Anatoli Shein commented on HDFS-11520: -- In the new patch I fixed the white-space issue and renamed 'callback' back to 'handler' to make it easier to review. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509798#comment-16509798 ] Anatoli Shein edited comment on HDFS-11520 at 6/12/18 4:05 PM: --- In the new patch I rebased and cleaned up the previous patch, added a test for canceling GetFileInfo RPC. was (Author: anatoli.shein): I the new patch I rebased and cleaned up the previous patch, added a test for canceling GetFileInfo RPC. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein reassigned HDFS-11520: Assignee: Anatoli Shein (was: Deepak Majeti) > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509798#comment-16509798 ] Anatoli Shein commented on HDFS-11520: -- I the new patch I rebased and cleaned up the previous patch, added a test for canceling GetFileInfo RPC. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Deepak Majeti >Priority: Major > Attachments: HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11520: - Attachment: HDFS-11520.trunk.001.patch > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Deepak Majeti >Priority: Major > Attachments: HDFS-11520.HDFS-8707.000.patch, > HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468798#comment-16468798 ] Anatoli Shein commented on HDFS-13534: -- [~James C], the new patch looks good and also builds fine for me: +1 (I think the previous 'override' error was just some setup weirdness on my end). > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch, HDFS-13534.001.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13534) libhdfs++: Fix GCC7 build
[
https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467504#comment-16467504
]
Anatoli Shein edited comment on HDFS-13534 at 5/8/18 4:01 PM:
--
[~James C], is the below line commented out on purpose?
{code:java}
npm install -g ember-cli{code}
Also, the following line is giving me an error "marked override, but does not
override":
{code:java}
ProducerResult Produce() override = 0;{code}
was (Author: anatoli.shein):
[~James C], is the below line commented out on purpose?
{code:java}
npm install -g ember-cli
{code}
> libhdfs++: Fix GCC7 build
> -
>
> Key: HDFS-13534
> URL: https://issues.apache.org/jira/browse/HDFS-13534
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
> Attachments: HDFS-13534.000.patch
>
>
> After merging HDFS-13403 [~pifta] noticed the build broke on some platforms.
> [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex
> implicitly included functional. Without that implicit include the compiler
> errors on the std::function in ioservice.h.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13534) libhdfs++: Fix GCC7 build
[
https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467504#comment-16467504
]
Anatoli Shein edited comment on HDFS-13534 at 5/8/18 2:52 PM:
--
[~James C], is the below line commented out on purpose?
{code:java}
npm install -g ember-cli
{code}
was (Author: anatoli.shein):
[~James C], is the below line commented out on purpose?
# npm install -g ember-cli
> libhdfs++: Fix GCC7 build
> -
>
> Key: HDFS-13534
> URL: https://issues.apache.org/jira/browse/HDFS-13534
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
> Attachments: HDFS-13534.000.patch
>
>
> After merging HDFS-13403 [~pifta] noticed the build broke on some platforms.
> [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex
> implicitly included functional. Without that implicit include the compiler
> errors on the std::function in ioservice.h.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467504#comment-16467504 ] Anatoli Shein commented on HDFS-13534: -- [~James C], is the below line commented out on purpose? # npm install -g ember-cli > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[
https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388044#comment-16388044
]
Anatoli Shein commented on HDFS-11807:
--
Thanks for another review, [~James C]!
In the new patch (009) I have addressed your comments as follows:
-check the return code on the socketpair() call
(/) Done.
-You can't assume a file descriptor is going to be in the range [0,255]; you'll
often get this for a process that has just spawned but that's just some luck.
If the upper bit is set it's also going to be explicitly sign extended when
it's promoted to a wider type which can make things a little confusing to
debug. I'd also convert the binary value to a string or hex encoding because if
the least significant byte of an int is 0 that's going to treated as a null
terminator.
{code:java}
// The argument contains child socket
fd[childsocket] = (int)argv[1][0];
{code}
(/) Fixed: I am passing it as a string now.
-Since we're sharing this test with libhdfs which can build on windows we can't
unconditionally include headers that windows won't have. Since this test also
leans heavily on *nix style process management e.g. fork() it might be best to
just avoid building this test on windows.
{code:java}
#include
{code}
(i) Agreed, eventually we might look into making this test windows compatible.
Nitpicky stuff, not blockers but would clean things up a little:
-Don't need an extra cast when calling free()
{code:java}
free((char*)httpHost);
{code}
(/) Done.
Same idea when writing httpHost over the socket
{code:java}
ASSERT_INT64_EQ(read(fd[parentsocket], (char*)httpHost, hostSize), hostSize);
{code}
(/) Done.
-I'd change "fd[]" so "fds" or something plural to make it's clear that it's an
array since a lot of C examples will name a single descriptor "fd". You can
also just make a single int variable at the top of main() that gets assigned
the appropriate side of the pair once you've forked just to avoid indexing
(you'll still need the array to pass to socketpair).
(/) Fixed: changed name from fd to fds.
> libhdfs++: Get minidfscluster tests running under valgrind
> --
>
> Key: HDFS-11807
> URL: https://issues.apache.org/jira/browse/HDFS-11807
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
>Priority: Major
> Attachments: HDFS-11807.HDFS-8707.000.patch,
> HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch,
> HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch,
> HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch,
> HDFS-11807.HDFS-8707.007.patch, HDFS-11807.HDFS-8707.008.patch,
> HDFS-11807.HDFS-8707.009.patch
>
>
> The gmock based unit tests generally don't expose race conditions and memory
> stomps. A good way to expose these is running libhdfs++ stress tests and
> tools under valgrind and pointing them at a real cluster. Right now the CI
> tools don't do that so bugs occasionally slip in and aren't caught until they
> cause trouble in applications that use libhdfs++ for HDFS access.
> The reason the minidfscluster tests don't run under valgrind is because the
> GC and JIT compiler in the embedded JVM do things that look like errors to
> valgrind. I'd like to have these tests do some basic setup and then fork
> into two processes: one for the minidfscluster stuff and one for the
> libhdfs++ client test. A small amount of shared memory can be used to
> provide a place for the minidfscluster to stick the hdfsBuilder object that
> the client needs to get info about which port to connect to. Can also stick
> a condition variable there to let the minidfscluster know when it can shut
> down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.009.patch > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch, > HDFS-11807.HDFS-8707.007.patch, HDFS-11807.HDFS-8707.008.patch, > HDFS-11807.HDFS-8707.009.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.008.patch > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch, > HDFS-11807.HDFS-8707.007.patch, HDFS-11807.HDFS-8707.008.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374396#comment-16374396 ] Anatoli Shein commented on HDFS-11807: -- Attached a patch just to poke the CI system. > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch, > HDFS-11807.HDFS-8707.007.patch, HDFS-11807.HDFS-8707.008.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12110) libhdfs++: Rebase 8707 branch onto an up to date version of trunk
[ https://issues.apache.org/jira/browse/HDFS-12110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-12110: - Resolution: Fixed Status: Resolved (was: Patch Available) > libhdfs++: Rebase 8707 branch onto an up to date version of trunk > - > > Key: HDFS-12110 > URL: https://issues.apache.org/jira/browse/HDFS-12110 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Deepak Majeti >Priority: Major > Attachments: HDFS-12110.HDFS-8707.000.patch, HDFS-12110.diff > > > It's been way too long since this has been done and it's time to start > knocking down blockers for merging into trunk. Can most likely just > copy/paste the libhdfs++ directory into a newer version of master. Want to > track it in a jira since it's likely to cause conflicts when pulling the > updated branch for the first time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12110) libhdfs++: Rebase 8707 branch onto an up to date version of trunk
[ https://issues.apache.org/jira/browse/HDFS-12110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367712#comment-16367712 ] Anatoli Shein commented on HDFS-12110: -- All maven and docker issues have been resolved, and rebase merged. Closing the issue. > libhdfs++: Rebase 8707 branch onto an up to date version of trunk > - > > Key: HDFS-12110 > URL: https://issues.apache.org/jira/browse/HDFS-12110 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Deepak Majeti >Priority: Major > Attachments: HDFS-12110.HDFS-8707.000.patch, HDFS-12110.diff > > > It's been way too long since this has been done and it's time to start > knocking down blockers for merging into trunk. Can most likely just > copy/paste the libhdfs++ directory into a newer version of master. Want to > track it in a jira since it's likely to cause conflicts when pulling the > updated branch for the first time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.007.patch Retrying since yetus failed. > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch, > HDFS-11807.HDFS-8707.007.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.006.patch Updated suppression file and added an Apache licence header. > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.005.patch In the new patch I added a suppression for the invalid free() error. The reason is __libc_freeres in older glibc versions causes this crash when linux hosts doesn't have IPv6 configured. More info here: http://valgrind.org/docs/manual/faq.html#faq.exit_errors > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.004.patch Whitespace fix > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.003.patch Thanks for the review, [~James C]. In the new patch I addressed your comments as follows: -stdlib is now included twice #include #include +#include (/) Fixed. -The EXPECT_EQ checks on return values are good, however in some cases if the EXPECT fails there's no chance the rest of the test will run correctly. For example if one of the write() calls fail it'd be better to just exit the test then let it produce what might be a confusing error. Easy fix would be to swap EXPECT_EQ with ASSERT_EQ for these cases. (/) Replaced EXPECT_EQ with ASSERT_EQ for all reads and writes. -I'd avoid using the hardcoded path "tmp" when you write the file you're going to send with curl. Instead check out what TempFile in configuration_test.h does to get a file name that's guaranteed to be unused. (/) Done. -libhdfs also uses this test so we don't want to hard code the libhdfspp_ prefix on API functions since that means this could only ever test libhdfs++. You can most likely apply the same trick that the libhdfspp_wrapper shims use to add the appropriate prefixes during the build. Alternatively I think you could build the binary to be used with valgrind with -DLIBHDFS_HDFS_H set so the shims aren't applied at all to avoid changing the calls. (/) Fixed by linking directly with hdfs.cc for this test (rather than going through the shim or hardcoding 'libhdfspp_' prefixes). -Might be worth checking the return value on snprintf to see if the string gets truncated. The paths returned by the temp file mechanism might be long enough to spill over 200 chars. (/) Done. > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[
https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anatoli Shein updated HDFS-11807:
-
Attachment: HDFS-11807.HDFS-8707.002.patch
In this patch I replaced getenv("USER") with getpwuid(), which should make it
more robust.
> libhdfs++: Get minidfscluster tests running under valgrind
> --
>
> Key: HDFS-11807
> URL: https://issues.apache.org/jira/browse/HDFS-11807
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
> Attachments: HDFS-11807.HDFS-8707.000.patch,
> HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch
>
>
> The gmock based unit tests generally don't expose race conditions and memory
> stomps. A good way to expose these is running libhdfs++ stress tests and
> tools under valgrind and pointing them at a real cluster. Right now the CI
> tools don't do that so bugs occasionally slip in and aren't caught until they
> cause trouble in applications that use libhdfs++ for HDFS access.
> The reason the minidfscluster tests don't run under valgrind is because the
> GC and JIT compiler in the embedded JVM do things that look like errors to
> valgrind. I'd like to have these tests do some basic setup and then fork
> into two processes: one for the minidfscluster stuff and one for the
> libhdfs++ client test. A small amount of shared memory can be used to
> provide a place for the minidfscluster to stick the hdfsBuilder object that
> the client needs to get info about which port to connect to. Can also stick
> a condition variable there to let the minidfscluster know when it can shut
> down.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.001.patch In this patch I fixed the problem with null-termination of a file descriptor when it is passed as an argument (which caused some hang-ups previously), added checks for the return values of reads and writes from/to the sockets, added the shutting down of the protobuf library (to avoid valgrind static data leaks), and added more comments. Currently it works on both my local machine and on docker. Please review. > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Attachment: HDFS-11807.HDFS-8707.000.patch First stab at running the mini dfs stress test under valgrind. > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > Attachments: HDFS-11807.HDFS-8707.000.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-11807: - Status: Patch Available (was: In Progress) libhdfs_mini_stress now runs under valgrind > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein reassigned HDFS-11807: Assignee: Anatoli Shein (was: James Clampffer) > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
