[jira] [Commented] (HDFS-14244) refactor the libhdfs++ build system
[ https://issues.apache.org/jira/browse/HDFS-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16773238#comment-16773238 ] James Clampffer commented on HDFS-14244: [~owen.omalley] anything that trims down the libhdfspp/third-party code sounds good to me. I tried applying the patch and ran into a couple issues doing the build in the container start_build_env.sh uses. -It looks like the dockerfile used by start_build_env.sh needs to install automake and autoconf. Doing a clean build in the container errors out when it tries to run those. After installing I make it to the next issue. -It looks like there's a dependency issue when building the whole tree at once. Things that use the minidfscluster wrapper code complain about missing symbols (e.g. nmdCreate). It looks like libhdfs.so has been built by the time it hits the error but cmake isn't picking up the library path while building tests. -I haven't made it far enough but it seems like the container will also need asio installed, or is this something the cmake external project approach handles? > refactor the libhdfs++ build system > --- > > Key: HDFS-14244 > URL: https://issues.apache.org/jira/browse/HDFS-14244 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs++, hdfs-client >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > > The current cmake for libhdfs++ has the source code for the dependent > libraries. By refactoring we can remove 150kloc of third party code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14047) [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c
[ https://issues.apache.org/jira/browse/HDFS-14047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16675067#comment-16675067 ] James Clampffer commented on HDFS-14047: Patch looks good to me, +1. Thanks for fixing this [~anatoli.shein]. > [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c > > > Key: HDFS-14047 > URL: https://issues.apache.org/jira/browse/HDFS-14047 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: libhdfs, native >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14047.000.patch, HDFS-14047.001.patch > > > Currently the native client CI tests break deterministically with these > errors: > Libhdfs > 1 - test_test_libhdfs_threaded_hdfs_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) > > Libhdfs++ > 34 - test_libhdfs_threaded_hdfspp_test_shim_static (Failed) > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:180 > with NULL return return value (errno: 2): expected substring: File does not > exist > [exec] TEST_ERROR: failed on > /testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c:336 > with return code -1 (errno: 2): got nonzero from doTestHdfsOperations(ti, > fs, ) > [exec] hdfsOpenFile(/tlhData0001/file1): > FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) > error: > [exec] (unable to get root cause for java.io.FileNotFoundException) > [exec] (unable to get stack trace for java.io.FileNotFoundException) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667461#comment-16667461 ] James Clampffer commented on HDFS-14033: Those two failures have been happening for a while in the native client test. Not related to this patch. > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14033.000.patch, HDFS-14033.001.patch > > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14033) [libhdfs++] Disable libhdfs++ build on systems that do not support thread_local
[ https://issues.apache.org/jira/browse/HDFS-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667363#comment-16667363 ] James Clampffer commented on HDFS-14033: The patch looks good, it should just skip the libhdfs++ build if the platform can't support the thread_local specifier. I don't have an environment set up right now that doesn't support thread_local but I was able to set THREAD_LOCAL_SUPPORTED to false after the check this patch adds in the CMakeList.txt and verify that libhdfs++ stopped building. I tested and this works fine on platforms that do support thread_local. Version numbers for compiler support in the message would be nice e.g. "GCC >= 4.8" but not a blocker for getting the release out. +1. > [libhdfs++] Disable libhdfs++ build on systems that do not support > thread_local > --- > > Key: HDFS-14033 > URL: https://issues.apache.org/jira/browse/HDFS-14033 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-14033.000.patch > > > In order to still be able to build Hadoop on older systems (such as rhel 6) > we need to disable libhdfs++ build on systems that do not support > thread_local. We should also emit a warning saying libhdfs++ wasn't built. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12134) libhdfs++: Add a synchronization interface for the GSSAPI
[
https://issues.apache.org/jira/browse/HDFS-12134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-12134:
---
Fix Version/s: HDFS-8707
> libhdfs++: Add a synchronization interface for the GSSAPI
> -
>
> Key: HDFS-12134
> URL: https://issues.apache.org/jira/browse/HDFS-12134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
> Fix For: HDFS-8707
>
> Attachments: HDFS-12134.HDFS-8707.000.patch,
> HDFS-12134.HDFS-8707.001.patch, HDFS-12134.HDFS-8707.002.patch,
> HDFS-12134.HDFS-8707.003.patch, HDFS-12134.HDFS-8707.004.patch
>
>
> Bits of the GSSAPI that Cyrus Sasl uses aren't thread safe. There needs to
> be a way for a client application to share a lock with this library in order
> to prevent race conditions. It can be done using event callbacks through the
> C API but we can provide something more robust (RAII) in the C++ API.
> Proposed client supplied lock, pretty much the C++17 lockable concept. Use a
> default if one isn't provided. This would be scoped at the process level
> since it's unlikely that multiple instances of libgssapi unless someone puts
> some effort in with dlopen/dlsym.
> {code}
> class LockProvider
> {
> virtual ~LockProvider() {}
> // allow client application to deny access to the lock
> virtual bool try_lock() = 0;
> virtual void unlock() = 0;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8707) Implement an async pure c++ HDFS client
[ https://issues.apache.org/jira/browse/HDFS-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665225#comment-16665225 ] James Clampffer commented on HDFS-8707: --- [~sunilg] I just took a look at HDFS-7285 since it has some similarities in the sense it was a longer term feature branch and it looked like the subtasks there use the parent jira as the fix version. Should we be consistent with that approach here? I have no strong preference either way, just want to be sure before updating everything. > Implement an async pure c++ HDFS client > --- > > Key: HDFS-8707 > URL: https://issues.apache.org/jira/browse/HDFS-8707 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Reporter: Owen O'Malley >Assignee: James Clampffer >Priority: Major > Fix For: 3.2.0 > > > As part of working on the C++ ORC reader at ORC-3, we need an HDFS pure C++ > client that lets us do async io to HDFS. We want to start from the code that > Haohui's been working on at https://github.com/haohui/libhdfspp . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12237) libhdfs++: PROTOC_IS_COMPATIBLE check fails if protobuf library is built from source
[ https://issues.apache.org/jira/browse/HDFS-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12237: --- Fix Version/s: 3.2.0 > libhdfs++: PROTOC_IS_COMPATIBLE check fails if protobuf library is built from > source > > > Key: HDFS-12237 > URL: https://issues.apache.org/jira/browse/HDFS-12237 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-12237.HDFS-8707.000.patch, > HDFS-12237.HDFS-8707.001.patch, HDFS-12237.HDFS-8707.002.patch > > > Looks like the PROTOC_IS_COMPATIBLE check fails when Protobuf library is > built from source. This happens because the check if performed during the > cmake phase, and the protobuf library needed for this test is build from > source only during the make phase, so the check fails with "ld: cannot find > -lprotobuf" because it was not built yet. We should probably restrict this > test to run only in cases when Protobuf library is already present and not > being built from source. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12427) libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine
[ https://issues.apache.org/jira/browse/HDFS-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12427: --- Fix Version/s: 3.2.0 > libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine > -- > > Key: HDFS-12427 > URL: https://issues.apache.org/jira/browse/HDFS-12427 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Fix For: 3.2.0 > > Attachments: HDFS-12427.HDFS-8707.000.patch, > HDFS-12427.HDFS-8707.001.patch, HDFS-12427.HDFS-8707.002.patch, > HDFS-12427.HDFS-8707.003.patch, HDFS-12427.HDFS-8707.004.patch > > > The lifetime of Request objects is tied to the worker thread(s) in the async > event loop. In the current code there's nothing that prevents a request from > outliving the RpcEngine (bound to FileSystem) while it's waiting for IO. If > the Request, or a task that makes a new request, outlives the RpcEngine it > attempts to dereference a dangling pointer and either crashes or continues to > run with bad data. > Proposed fix is to reference count the RpcEngine via shared_ptr so that > Requests can hold a weak_ptr to it. When a request or RpcConnection > attempting to make a request needs something from the RpcEngine like a call > id number it can promote the weak_ptr to a shared_ptr. If it's unable to > promote because the RpcEngine has been destroyed the Request's handler can be > invoked with an appropriate error message. A weak_ptr must be used rather > than a shared_ptr to avoid reference cycles. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10686) libhdfs++: implement delegation token authorization
[ https://issues.apache.org/jira/browse/HDFS-10686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10686: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: implement delegation token authorization > --- > > Key: HDFS-10686 > URL: https://issues.apache.org/jira/browse/HDFS-10686 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: James Clampffer >Priority: Major > > The current libhdfs++ SASL implementation does a kerberos handshake for each > connection. HDFS includes support for issuing and using time-limited tokens > to reduce the load on the kerberos server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10611) libhdfs++: Add support for HA configurations with more than 2 namenodes
[ https://issues.apache.org/jira/browse/HDFS-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10611: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Add support for HA configurations with more than 2 namenodes > --- > > Key: HDFS-10611 > URL: https://issues.apache.org/jira/browse/HDFS-10611 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > > Placeholder for now, doesn't look like you can use more than two namenodes > per nameservice at the moment. It shouldn't be too hard to extend > HANamenodeTracker to support this in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11388) libhdfs++: Add an alternative to assert()
[ https://issues.apache.org/jira/browse/HDFS-11388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11388: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Add an alternative to assert() > - > > Key: HDFS-11388 > URL: https://issues.apache.org/jira/browse/HDFS-11388 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Priority: Minor > > There's a few places we use C's assert, in a lot of applications it would be > better to switch to something that doesn't necessarily call abort() right > away. > Example would be a large program that uses temp files or is responsible for > freeing shared memory used for IPC. Aborting the process will clean up > resources private to the process but leave other sorts of resources hanging > around. If assert is replaced by a different macro it could throw or signal > to indicate that it's no longer safe to use the library but give the rest of > the application a chance to clean up unrelated resources. > I'd also be nice to have a variant that wasn't compiled out on release builds > for inexpensive checks. There have been issues in the past where even -O2 > optimized release builds were able to collapse templates/inline/devirtualize > things such that code that looked like it had been working started acting > weird (was relying on UB) and catching that with existing checks would be > valuable. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11437) libhdfs++: Handler for FileSystem async connect can be invoked before successful communication with active NN
[ https://issues.apache.org/jira/browse/HDFS-11437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11437: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Handler for FileSystem async connect can be invoked before > successful communication with active NN > - > > Key: HDFS-11437 > URL: https://issues.apache.org/jira/browse/HDFS-11437 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > > The handler provided to FileSystem::Connect can be invoked as soon as the FS > makes a connection to the standby NN rather than waiting until it connects to > the active NN. This allows RPC requests to be enqueued before a real > connection is made and if the active NN isn't reachable for some reason the > only way to cancel is to delete the FS from another thread which kills all > pending requests. > The underlying issue is that currently the only thing that must happen for > the connect handler to be invoked is a successful handshake with a NN. > Connecting to the standby NN and receiving a StandbyException satisfies this > requirement but it should wait until it is able to get a handshake from the > active NN. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11556) libhdfs++: Improve the usage of std::promise and std::future for sync shims
[
https://issues.apache.org/jira/browse/HDFS-11556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-11556:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: Improve the usage of std::promise and std::future for sync shims
> ---
>
> Key: HDFS-11556
> URL: https://issues.apache.org/jira/browse/HDFS-11556
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
>
> There's two issues with the current promise/future situation being used in
> the synchronous shims.
> 1) C++ std::promise and std::futures aren't particularly intuitive (at least
> for me). Logically it makes sense to think of the promise and future objects
> as handles that each hold onto a shared object containing the promised type +
> a mutex and condition variable for synchronization. When promise::set(T
> val) is called it seems reasonable to expect that it writes val into the
> shared state, and and wakes the thread blocked in future<>::wait. This would
> be standard conformant but generally isn't how it's implemented in the
> standard libraries.
> The common implementation is to bundle the value, mutex, and cond_var into
> the promise. This makes it really easy to introduce race conditions that
> will often pass tests.. until they don't. Example of most common case:
> {code}
> auto foo = std::promise()
> std::thread async_task([](){
> ... do some work ...
> foo.set_value(1);
> })
> auto future_val = foo.get_future();
> int result = future_val.get_value();
> {code}
> That last line has a race between promise::~promise() which happens right
> after it gets set and any reads of the value, mutex, or condition variable
> owned by the promise but referenced by the future.
> When it does show up there will typically be a thread hung in
> pthread_cond_wait and an invalid 32 bit read of the futex state that the
> condition variable used under the hood.
> 2) If a callback happens to throw in an operation that has synchronous shims
> on top of it the stack will unwind and io_service.run_once will be called to
> keep getting work done (HDFS-9699). The problem here is that once the stack
> is unwound there's nothing around to set the promise so the thread blocked on
> a future will never wake up. The simple/unrealistic fix is to assert that
> exceptions should never be used but that precludes the use of this library in
> most C++ projects and the use of most third party dependencies that weren't
> made by Google.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12724) Add retry logic when data node block moves while reading
[ https://issues.apache.org/jira/browse/HDFS-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12724: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > Add retry logic when data node block moves while reading > > > Key: HDFS-12724 > URL: https://issues.apache.org/jira/browse/HDFS-12724 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Deepak Majeti >Assignee: Deepak Majeti >Priority: Major > Attachments: HDFS-12724.HDFS-8707.000.patch > > > It is possible for a data node block to move due to re-balance. > Libhdfs++ must try a replica block when a block re-balance happens while > reading data from that block. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11973) libhdfs++: Remove redundant directories in examples
[ https://issues.apache.org/jira/browse/HDFS-11973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11973: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Remove redundant directories in examples > --- > > Key: HDFS-11973 > URL: https://issues.apache.org/jira/browse/HDFS-11973 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11973.HDFS-8707.000.patch, > HDFS-11973.HDFS-8707.001.patch, HDFS-11973.HDFS-8707.002.patch > > > In order to keep consistent with the tools and tests I think we should remove > one level of directories in examples folder. > E.g this directory: > /hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/cat/cat.c > Should become this: > /hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/cat.c > Removing the redundant directories will also simplify our cmake file > maintenance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11380) libhdfs++: Extend logger to optionally print stack traces
[ https://issues.apache.org/jira/browse/HDFS-11380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11380: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Extend logger to optionally print stack traces > -- > > Key: HDFS-11380 > URL: https://issues.apache.org/jira/browse/HDFS-11380 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Minor > Attachments: HDFS-11380.HDFS-8707.000.patch > > > It'd be useful if the logger could optionally print a backtrace along with > the log message. Something using libc's backtrace/backtrace_symbols which > would be portable and provide enough info for narrowing down most problems > quickly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11884) libhdfs++: Make use of socket wrappers consistent
[ https://issues.apache.org/jira/browse/HDFS-11884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11884: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Make use of socket wrappers consistent > - > > Key: HDFS-11884 > URL: https://issues.apache.org/jira/browse/HDFS-11884 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > > Different areas of the code alternate between hdfs::async_stream (and > derivatives) and the more direct asio::ip::tcp::socket. I propose getting > the code to consistently do socket IO through the async_stream style classes. > -async_stream's async_read and async_write are virtual so we can get rid of > classes that template the socket type to support trait injection for mock > tests > -async_stream can have debug info attached to it e.g. a human readable host > and port. Extra info about status of the stream clears the way for > implementing a DN connection cache + bookkeeping how many sockets are in use. > -gives a chance to have a mutex serialize read and write operations. Not > needed in current use cases but can be used to assert that there aren't > concurrent reads and writes to the same socket happening unintentionally -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12276) libhdfs++: uri parser has clang warnings that break external projects
[
https://issues.apache.org/jira/browse/HDFS-12276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-12276:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: uri parser has clang warnings that break external projects
> -
>
> Key: HDFS-12276
> URL: https://issues.apache.org/jira/browse/HDFS-12276
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
>Priority: Major
>
> When trying to do a clang build of libhdfs++ as part of the ORC project I get
> some warnings that are treated as errors:
> In file included from
> /home/travis/build/apache/orc/c++/libs/libhdfspp/include/hdfspp/options.h:21:
> /home/travis/build/apache/orc/c++/libs/libhdfspp/include/hdfspp/uri.h:30:7:
> error: 'uri_parse_error' has no out-of-line virtual method definitions; its
> vtable will be emitted in every translation unit [-Werror,-Wweak-vtables]
> class uri_parse_error : public std::invalid_argument {
> ^
> 2 errors generated.
> These should be fixed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10543) hdfsRead read stops at block boundary
[
https://issues.apache.org/jira/browse/HDFS-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10543:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> hdfsRead read stops at block boundary
> -
>
> Key: HDFS-10543
> URL: https://issues.apache.org/jira/browse/HDFS-10543
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Xiaowei Zhu
>Assignee: James Clampffer
>Priority: Major
> Fix For: HDFS-8707
>
> Attachments: HDFS-10543.HDFS-8707.000.patch,
> HDFS-10543.HDFS-8707.001.patch, HDFS-10543.HDFS-8707.002.patch,
> HDFS-10543.HDFS-8707.003.patch, HDFS-10543.HDFS-8707.004.patch
>
>
> Reproducer:
> char *buf2 = new char[file_info->mSize];
> memset(buf2, 0, (size_t)file_info->mSize);
> int ret = hdfsRead(fs, file, buf2, file_info->mSize);
> delete [] buf2;
> if(ret != file_info->mSize) {
> std::stringstream ss;
> ss << "tried to read " << file_info->mSize << " bytes. but read " <<
> ret << " bytes";
> ReportError(ss.str());
> hdfsCloseFile(fs, file);
> continue;
> }
> When it runs with a file ~1.4GB large, it will return an error like "tried to
> read 146890 bytes. but read 134217728 bytes". The HDFS cluster it runs
> against has a block size of 134217728 bytes. So it seems hdfsRead will stop
> at a block boundary. Looks like a regression. We should add retry to continue
> reading cross blocks in case of files w/ multiple blocks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10781) libhdfs++: redefine NN timeout to be "time without a response"
[ https://issues.apache.org/jira/browse/HDFS-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10781: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: redefine NN timeout to be "time without a response" > -- > > Key: HDFS-10781 > URL: https://issues.apache.org/jira/browse/HDFS-10781 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > > In the find tool, we submit a zillion requests to the NameNode > asynchronously. As the queue on the NameNode grows, the time to response for > each individual message will increase. In the find tool, we were eventually > getting timeouts on requests, even though the NN was respoinding as fast as > its little feet could carry it. > I propose that we should redefine timeouts to be on a per-connection basis > rather than per-request. If a client has an outstanding request to the NN > but hasn't gotten a response back within n msec, it should declare the > connection dead and retry. As long as the NameNode is being responsive to > the best of its ability and providing data, we will not declare the link dead. > One potential for Failure of Least Astonishment here is that it will mean any > particular request from a client cannot be depended on to get a positive or > negative response within a fixed amount of time, but I think that may be a > good trade to make. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10951) libhdfs++: FileHandle should handle DN failure internally
[ https://issues.apache.org/jira/browse/HDFS-10951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10951: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: FileHandle should handle DN failure internally > - > > Key: HDFS-10951 > URL: https://issues.apache.org/jira/browse/HDFS-10951 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Minor > > Currently if a read hits a bad DN it will add it to the DN exclusion set and > return an error. It'd be nice if it'd exhaust attempts to all replica > locations before returning an error so client applications don't have to > attempt to figure out the difference between "File not found" and "DN not > responsive, go try a different DN". > Related to HDFS-10937 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12168) libhdfs++: test4tests doesn't detect C++ test changes
[ https://issues.apache.org/jira/browse/HDFS-12168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12168: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: test4tests doesn't detect C++ test changes > - > > Key: HDFS-12168 > URL: https://issues.apache.org/jira/browse/HDFS-12168 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Priority: Minor > > All patches "fail" because test4tests doesn't look for new or updated C++ > tests. This isn't a huge deal as long as everyone is careful with their > patches and reviews, but I think this would be really nice to have prior to > integrating into the mainline branch. > Not sure if it's worth doing before rebasing onto a newer version of trunk in > case there's any build system changes that'd break the patch. I'd be > thrilled if anyone who knows the maven and CI infrastructure well could give > some pointers about how to go about extending the test framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12640) libhdfs++: automatic CI tests are getting stuck in test_libhdfs_mini_stress_hdfspp_test_shim_static
[ https://issues.apache.org/jira/browse/HDFS-12640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12640: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: automatic CI tests are getting stuck in > test_libhdfs_mini_stress_hdfspp_test_shim_static > --- > > Key: HDFS-12640 > URL: https://issues.apache.org/jira/browse/HDFS-12640 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-12640.HDFS-8707.000.patch > > > All of the automated tests seem to get stuck, or at least stop generating > useful output, in test_libhdfs_mini_stress_hdfspp_test_shim_static. Not able > to reproduce the issue locally in docker. > Right now this is blocking a few patches, and not having those patches > committed is slowing down work on other parts of the library. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12111) libhdfs++: Expose HA and Kerberos options for C++ minidfscluster bindings
[ https://issues.apache.org/jira/browse/HDFS-12111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer reassigned HDFS-12111: -- Assignee: (was: James Clampffer) > libhdfs++: Expose HA and Kerberos options for C++ minidfscluster bindings > - > > Key: HDFS-12111 > URL: https://issues.apache.org/jira/browse/HDFS-12111 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Priority: Major > > Provide an easy way to instantiate the hdfs::MiniCluster object with HA > and/or Kerberos enabled. The majority of the existing CI tests should be > able to run in those environments and a few HA and Kerberos smoke tests can > be added as part of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12111) libhdfs++: Expose HA and Kerberos options for C++ minidfscluster bindings
[ https://issues.apache.org/jira/browse/HDFS-12111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12111: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Expose HA and Kerberos options for C++ minidfscluster bindings > - > > Key: HDFS-12111 > URL: https://issues.apache.org/jira/browse/HDFS-12111 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Priority: Major > > Provide an easy way to instantiate the hdfs::MiniCluster object with HA > and/or Kerberos enabled. The majority of the existing CI tests should be > able to run in those environments and a few HA and Kerberos smoke tests can > be added as part of this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11520: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10937) libhdfs++: hdfsRead return -1 at eof
[
https://issues.apache.org/jira/browse/HDFS-10937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10937:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: hdfsRead return -1 at eof
>
>
> Key: HDFS-10937
> URL: https://issues.apache.org/jira/browse/HDFS-10937
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Bob Hansen
>Priority: Major
>
> The libhdfs++ implementation of hdfsRead appears to be out-of-spec. The
> header says it will return 0 at eof, but the current implementation returns
> -1 with an errno of 261 (invalid offset).
> The basic posix-y read loop of
> while ( (bytesRead = hdsfRead(...)) != 0) {...}
> won't work with with libhdfs++'s hdfsRead method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11544) libhdfs++: Improve C API error reporting
[ https://issues.apache.org/jira/browse/HDFS-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11544: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Improve C API error reporting > > > Key: HDFS-11544 > URL: https://issues.apache.org/jira/browse/HDFS-11544 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11544.HDFS-8707.000.patch, > HDFS-11544.HDFS-8707.001.patch, HDFS-11544.HDFS-8707.002.patch > > > The thread local string used for hdfsGetLastError wasn't reset between calls > so it could give stale results in confusing ways. Now it gets reset with a > placeholder that says that hasn't set an error string. > Also fixed indentation that wasn't consistent + marked which functions are > used by the hdfs.h API and hdfs_ext.h API to make it easier to see when > changes could break compatibility. Included some minor cleanup for the > common case catch blocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10790) libhdfs++: split recursive versions of SetPermission and SetOwner to SetAllPermissions and SetAllOwner
[ https://issues.apache.org/jira/browse/HDFS-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10790: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: split recursive versions of SetPermission and SetOwner to > SetAllPermissions and SetAllOwner > -- > > Key: HDFS-10790 > URL: https://issues.apache.org/jira/browse/HDFS-10790 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > > We currently have a flag that we pass in to SetPermission and SetOwner that > change the semantics of the call. We should split it into two functions, one > that does an efficient, direct version, and the other that does globbing and > optionally recursion. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10596) libhdfs++: Implement hdfsFileIsEncrypted
[ https://issues.apache.org/jira/browse/HDFS-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10596: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Implement hdfsFileIsEncrypted > > > Key: HDFS-10596 > URL: https://issues.apache.org/jira/browse/HDFS-10596 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Priority: Major > Attachments: HDFS-10596.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10836) libhdfs++: Add a NO_TOOLS and NO_EXAMPLES flag to cmake
[ https://issues.apache.org/jira/browse/HDFS-10836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10836: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Add a NO_TOOLS and NO_EXAMPLES flag to cmake > --- > > Key: HDFS-10836 > URL: https://issues.apache.org/jira/browse/HDFS-10836 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > > For some instances, our consumers will want just the library, and not want to > compile (and filgure out linking) for the tools and examples. Let's add a > CMake flag to turn those off. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10952) libhdfs++: Fix test_libhdfs_threaded_hdfspp_test_shim_static non-deterministic failures
[ https://issues.apache.org/jira/browse/HDFS-10952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10952: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Fix test_libhdfs_threaded_hdfspp_test_shim_static > non-deterministic failures > --- > > Key: HDFS-10952 > URL: https://issues.apache.org/jira/browse/HDFS-10952 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > > test_libhdfs_threaded_hdfspp_test_shim_static has started failing more often > and not a whole lot has changed that should directly impact this. > It looks like the failure mode is error injection in the read pipeline. > Until HDFS-10951 is completed reads don't have a retry mechanism so any error > injection will cause that read to fail. However that doesn't explain why > these failures have been happening more frequently lately. The failures seem > to be hardware or timing specific; they almost never happen in some > environments and happen fairly frequently in others. It might help to set up > a test harness to step through revisions and see if a specific commit causes > the probability of a failure to go up. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10621) libhdfs++: Implement . (dot) and .. (double-dot) semantics
[ https://issues.apache.org/jira/browse/HDFS-10621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10621: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Implement . (dot) and .. (double-dot) semantics > -- > > Key: HDFS-10621 > URL: https://issues.apache.org/jira/browse/HDFS-10621 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Priority: Major > > We need to implement . (dot) and .. (double-dot) semantics in hdfs.cc in > getAbsolutePath, hdfsSetWorkingDirectory, hdfsGetWorkingDirectory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10607) libhdfs++:hdfs_shim missing some hdfs++ functions
[ https://issues.apache.org/jira/browse/HDFS-10607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10607: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++:hdfs_shim missing some hdfs++ functions > - > > Key: HDFS-10607 > URL: https://issues.apache.org/jira/browse/HDFS-10607 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > > The hdfsConfGetStr, hdfsConfGetInt, hdfsStrFree, hdfsSeek, and hdfsTell > functions are all calling into the libhdfs implementations, not the libhdfs++ > implementations. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10492) libhdfs++: Clean up minidfscluster tests
[ https://issues.apache.org/jira/browse/HDFS-10492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10492: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Clean up minidfscluster tests > > > Key: HDFS-10492 > URL: https://issues.apache.org/jira/browse/HDFS-10492 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > > A couple things need to be fixed with the minidfscluster tests > -Tests that are using hdfs_ext shouldn't be living in libhdfs-tests, any test > in there should be able to be shared between both libraries. > -Tests added in HDFS-9890 relies on NDEBUG to turn on error simulation, > ideally these should have some other switch so we can run error simulation on > release builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11206) libhdfs++: FileSystem doesn't handle directory paths with a trailing "/"
[ https://issues.apache.org/jira/browse/HDFS-11206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11206: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: FileSystem doesn't handle directory paths with a trailing "/" > > > Key: HDFS-11206 > URL: https://issues.apache.org/jira/browse/HDFS-11206 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Trivial > > FileSystem methods that expect directories error when they receive a path > with a trailing slash. The java hadoop CLI tool is able to handle this > without any issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10634) libhdfs++: Improve parsing of config file entries
[ https://issues.apache.org/jira/browse/HDFS-10634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10634: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Improve parsing of config file entries > - > > Key: HDFS-10634 > URL: https://issues.apache.org/jira/browse/HDFS-10634 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > > Config files just specify an authority rather than a real URI for namenodes, > but we've been using the URI class to parse them. This is kind of hacky > because a scheme needs to be prepended (and then ignored) for the library to > work. > The URI parsing library generates errors in valgrind when it doesn't get a > scheme which could be concerning (conditional jump on undefined). At the > moment it's unclear if this is a real issue or it's just using vectorized > string operations that read whole words but the string ends in the middle of > a machine word. > This is also a good place to refactor the split function in uri.h to be > general purpose, right now it has a special rule to disregard leading '/' > chars. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10513) libhdfs++: Ensure we're connected when FS methods are called
[ https://issues.apache.org/jira/browse/HDFS-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10513: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Ensure we're connected when FS methods are called > > > Key: HDFS-10513 > URL: https://issues.apache.org/jira/browse/HDFS-10513 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Priority: Major > > We can either transparently connect or complain if we're not -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10513) libhdfs++: Ensure we're connected when FS methods are called
[ https://issues.apache.org/jira/browse/HDFS-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665146#comment-16665146 ] James Clampffer commented on HDFS-10513: [~anatoli.shein] Can you add more to the description here? Is this talking about attempting to do RPC before FileSystem::Connect has been called or making sure a NN RPC connection is reopened if there's been an error between the Connect call and when RPC needs to be sent later on? > libhdfs++: Ensure we're connected when FS methods are called > > > Key: HDFS-10513 > URL: https://issues.apache.org/jira/browse/HDFS-10513 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Priority: Major > > We can either transparently connect or complain if we're not -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10451) libhdfs++: Look up kerberos principal by username
[ https://issues.apache.org/jira/browse/HDFS-10451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10451: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Look up kerberos principal by username > - > > Key: HDFS-10451 > URL: https://issues.apache.org/jira/browse/HDFS-10451 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > > SaslProtocol::Negotiate passes the user name directly to the sasl_engine for > authentication; the SASL engines require that. > HDFS maps princpals to usernames by stripping off the realm and hostname. We > should query the ccache for all available tickets, and find the one that best > matches the passed-in username using the HDFS semantics. e.g. if the > username is client1, and we have a ticket for > client1/machine1.foo@foo.com, we should use that ticket. > If multiple tickets match, the one that most exactly matches the username > (host, realm) should be used. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10487) libhdfs++: support fs.default.name
[ https://issues.apache.org/jira/browse/HDFS-10487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10487: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: support fs.default.name > -- > > Key: HDFS-10487 > URL: https://issues.apache.org/jira/browse/HDFS-10487 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > > Libhdfs++ connects to the value specified by fs.defaultFS in the > configuration file. > Some older configurations may be using fs.default.name instead. Either > should be accepted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10411) libhdfs++: Incorrect parse of URIs from hdfs-site.xml
[
https://issues.apache.org/jira/browse/HDFS-10411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10411:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: Incorrect parse of URIs from hdfs-site.xml
> --
>
> Key: HDFS-10411
> URL: https://issues.apache.org/jira/browse/HDFS-10411
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: James Clampffer
>Priority: Major
>
> It looks like the URI class confuses the host and scheme if the original URI
> didn't have a scheme.
> Example from hdfs-site.xml. Config generated using the cloudera MC.
> {code}
> dfs.namenode.servicerpc-address.nameservice1.namenode86
> this-is-node-01.duder.com:8022
> {code}
> host = empty string
> port = unset optional
> scheme = this-is-node-01.duder.com
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12169) libhdfs++: Exceptions from third party libs aren't caught
[ https://issues.apache.org/jira/browse/HDFS-12169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-12169: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Exceptions from third party libs aren't caught > - > > Key: HDFS-12169 > URL: https://issues.apache.org/jira/browse/HDFS-12169 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Priority: Major > > Some of our third party libraries throw and it's unclear if we properly catch > exceptions in the places they need to be dealt with. We should do a pass > over the public API of each library and catch exceptions close to the calls > that throw them. Right now the async worker threads make a last-ditch effort > to prevent them from exiting the library but there's situations where RAII > hasn't been done right (fixing this is critical too) and unwinding the stack > leaves things in an inconsistent state. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10704) libhdfs++: FileSystem should not take ownership of IOService
[ https://issues.apache.org/jira/browse/HDFS-10704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10704: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: FileSystem should not take ownership of IOService > > > Key: HDFS-10704 > URL: https://issues.apache.org/jira/browse/HDFS-10704 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > > The ctor for the filesystem currently takes ownership of the IOService passed > in. There is a valid use case for a single IOService for multiple > FileSystems (e.g. for different users). > If an IOService is passed in, the consumer should be responsible for its > lifetime. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11266) libhdfs++: Redesign block reader with with simplicity and resource management in mind
[
https://issues.apache.org/jira/browse/HDFS-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-11266:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: Redesign block reader with with simplicity and resource management
> in mind
> -
>
> Key: HDFS-11266
> URL: https://issues.apache.org/jira/browse/HDFS-11266
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
> Attachments: HDFS-11266.HDFS-8707.000.patch
>
>
> The goal here is to significantly simplify the block reader and make it much
> harder to introduce issues. There are plenty of examples of these issues in
> the subtasks of HDFS-8707, the one that finally motivated a reimplementation
> is HDFS-10931.
> Goals:
> -The read side protocol of the data transfer pipeline is fundamentally really
> simple (even if done asynchronously). The code should be equally simple.
> -Get the code in a state that should be easy enough to reason about with a
> solid understanding of HDFS and basic understanding of C++ and vice versa:
> improve comments and avoid using esoteric C++ constructs. This is a
> must-have in order to lower the bar to contribute.
> -Get rid of dependencies on the existing continuation stuff. Myself and
> others have spent far too much time debugging both the continuation code and
> bugs introduced because the continuation code was hard to reason about.
> Notable issues:
> -It's cool from a theoretical perspective, but after 18 months of working
> on this it's still unclear what problem the continuation pattern helped solve
> that callbacks couldn't.
> -They spend more time allocating memory than the rest of the code does
> doing real work - seriously, profile it. This can't be fixed because the
> Pipeline takes ownership of all Continuation objects and then deletes them.
> -The way the block reader really uses them is a hybrid of a state machine,
> continuations, and directly using asio callbacks to bounce between the two.
> Proposed approach:
> Still have a BlockReader class that owns a PacketReader class, the packet
> reader is analogous to the ReadPacketContinuation that the BlockReader builds
> now. The difference is that none of this will be stitched together at
> runtime using continuations, and once we have a block reader with a member
> packet reader that gets allocated up front. The PacketReader can be recycled
> in order to avoid allocations. The block reader is only responsible for
> requesting block info, after that it keeps invoking the PacketReader until
> enough data has been read.
> Async chaining:
> Move to a state machine based approach. This allows the readers to be pinned
> in memory, where each state is represented as a method. The asynchronous IO
> becomes the state transitions. A callback is supplied to the asio async call
> that jumps to the next state upon completion of the IO operation. Epsilon
> transitions will be fairly rare, but if we need them to temporarily drop a
> lock as is done in the RPC code io_service::post can be used rather than a
> call that actually does IO.
> I'm fairly confident in this approach since I used the same to implement
> various hardware async bus interfaces in VHDL to good effect i.e. high
> performance and easy to understand. An asio callback is roughly analogous to
> a signal in a sensitivity list as the methods are to process blocks.
> Example state machine that would send some stuff, then wait to get something
> back like what the current BlockReader::AsyncRequestBlock does using the
> approach described above.
> {code}
> class ExampleHandshake {
> // class would own any small buffers so they can be directly accessed
> public:
> void SendHandshake();
> private:
> void OnHandshakeSend();
> void OnHandShakeDone();
> asio::io_service service_;
> asio::ip::tcp::socket socket_;
> }
> void ExampleHandshake::SendHandshake() {
> // trampoline to jump into read state once write completes
> auto trampoline[this](asio::error_code ec, size_t sz) {
> //error checking here
>this->OnHandshakeSend();
> };
> asio::write(service_, socket_, asio buffer of data here, trampoline);
> }
> void ExampleHandshake::OnHandshakeSend() {
> // when read completes bounce into handler
> auto trampoline = [this](asio::error_code ec, size_t sz) {
> this->OnHandshakeDone();
> };
> asio::read(service_, socket_, asio buffer for received data, trampoline);
> }
> void ExampleHandshake::OnHandshakeDone() {
> //just finished sending request, and receiving response, go do something
> }
> {code}
--
This message was sent by Atlassian JIRA
[jira] [Updated] (HDFS-10554) libhdfs++: signed to unsigned conversions are breaking things and compiler isn't issuing expected warnings
[
https://issues.apache.org/jira/browse/HDFS-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10554:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: signed to unsigned conversions are breaking things and compiler
> isn't issuing expected warnings
> --
>
> Key: HDFS-10554
> URL: https://issues.apache.org/jira/browse/HDFS-10554
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
>
> There's at least two places where we use -1 to indicate unset/default values
> that end up getting cast into unsigned integers. The compiler should be
> smart enough to figure this out and issue a warning but it's not; we need to
> find out what's going on there. We also need to fix the places where this
> sort of thing has found its way into the code:
> In URI
> {code}
> // -1 if the port is undefined.
> optional get_port() const
> { return port; }
> {code}
> In Options (gets cast to uint64_t somewhere)
> {code}
> /**
> * Maximum number of retries for RPC operations
> **/
> int max_rpc_retries;
> static const int kNoRetry = -1;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10380) libhdfs++: Get rid of socket template parameter in RpcConnection
[ https://issues.apache.org/jira/browse/HDFS-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10380: --- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Get rid of socket template parameter in RpcConnection > - > > Key: HDFS-10380 > URL: https://issues.apache.org/jira/browse/HDFS-10380 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10380.HDFS-8707.000.patch > > > RpcConnection is always templated on asio::ip::tcp::socket except for in > rpc_engine_test.cc. My understanding is the original reason for using this > as a template parameter was to be able to use trait injection in gmock tests. > This is useful for testing but makes debugging a lot more tricky due to the > extra stuff that shows up on the stack. Heavily templated code also tends to > produce very unhelpful compile errors e.g. when a missing semicolon in a > templated class turns into pages of errors coming out of stuff that depended > on the instantiation. > This sort of work was already accomplished elsewhere by HDFS-9144, it looks > like RpcConnection was one of the few areas they were left in place. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10355) Fix thread_local related build issue on Mac OS X
[
https://issues.apache.org/jira/browse/HDFS-10355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10355:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> Fix thread_local related build issue on Mac OS X
>
>
> Key: HDFS-10355
> URL: https://issues.apache.org/jira/browse/HDFS-10355
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
> Environment: OS: Mac OS X 10.11
> clang: Apple LLVM version 7.0.2 (clang-700.1.81)
>Reporter: Tibor Kiss
>Assignee: James Clampffer
>Priority: Major
> Attachments: HDFS-10355.HDFS-8707.000.patch
>
>
> The native hdfs library uses C++11 features heavily.
> One of such feature is thread_local storage class which is supported in GCC,
> Visual Studio and the community version of clang compiler, but not by Apple's
> clang (which is default on OS X boxes).
> See further details here: http://stackoverflow.com/a/29929949
> Even though not many Hadoop cluster runs on OS X developers still use this
> platform for development.
> The problem can be solved multiple ways:
> a) Stick to gcc/g++ or community based clang on OS X. Developers will need
> extra steps to build Hadoop.
> b) Workaround thread_local with a helper class.
> c) Get rid of all the globals marked with thread_local. Interface change
> will be erquired.
> d) Disable multi threading support in the native client on OS X and document
> this limitation.
> Compile error related to thread_local:
> {noformat}
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc:66:1:
> error: thread-local storage is not supported for the current target
> [exec] thread_local std::string errstr;
> [exec] ^
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc:87:1:
> error: thread-local storage is not supported for the current target
> [exec] thread_local std::experimental::optional
> fsEventCallback;
> [exec] ^
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc:88:1:
> error: thread-local storage is not supported for the current target
> [exec] thread_local std::experimental::optional
> fileEventCallback;
> [exec] ^
> [exec] 1 warning and 3 errors generated.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10354) Fix compilation & unit test issues on Mac OS X with clang compiler
[
https://issues.apache.org/jira/browse/HDFS-10354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10354:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> Fix compilation & unit test issues on Mac OS X with clang compiler
> --
>
> Key: HDFS-10354
> URL: https://issues.apache.org/jira/browse/HDFS-10354
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
> Environment: OS X: 10.11
> clang: Apple LLVM version 7.0.2 (clang-700.1.81)
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Major
> Attachments: HDFS-10354.HDFS-8707.001.patch,
> HDFS-10354.HDFS-8707.002.patch
>
>
> Compilation fails with multiple errors on Mac OS X.
> Unit test test_test_libhdfs_zerocopy_hdfs_static also fails to execute on OS
> X.
> Compile error 1:
> {noformat}
> [exec] Scanning dependencies of target common_obj
> [exec] [ 45%] Building CXX object
> main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/base64.cc.o
> [exec] [ 45%] Building CXX object
> main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/status.cc.o
> [exec] [ 46%] Building CXX object
> main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/sasl_digest_md5.cc.o
> [exec] [ 46%] Building CXX object
> main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/hdfs_public_api.cc.o
> [exec] [ 47%] Building CXX object
> main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/options.cc.o
> [exec] [ 48%] Building CXX object
> main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/configuration.cc.o
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration.cc:85:12:
> error: no viable conversion from 'optional' to 'optional'
> [exec] return result;
> [exec]^~
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/tr2/optional.hpp:427:13:
> note: candidate constructor not viable: no known conversion from
> 'std::experimental::optional' to 'std::experimental::nullopt_t' for 1st
> argument
> [exec] constexpr optional(nullopt_t) noexcept : OptionalBase() {};
> [exec] ^
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/tr2/optional.hpp:429:3:
> note: candidate constructor not viable: no known conversion from
> 'std::experimental::optional' to 'const
> std::experimental::optional &' for 1st argument
> [exec] optional(const optional& rhs)
> [exec] ^
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/tr2/optional.hpp:438:3:
> note: candidate constructor not viable: no known conversion from
> 'std::experimental::optional' to 'std::experimental::optional long> &&' for 1st argument
> [exec] optional(optional&& rhs)
> noexcept(is_nothrow_move_constructible::value)
> [exec] ^
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/tr2/optional.hpp:447:13:
> note: candidate constructor not viable: no known conversion from
> 'std::experimental::optional' to 'const long long &' for 1st argument
> [exec] constexpr optional(const T& v) : OptionalBase(v) {}
> [exec] ^
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/tr2/optional.hpp:449:13:
> note: candidate constructor not viable: no known conversion from
> 'std::experimental::optional' to 'long long &&' for 1st argument
> [exec] constexpr optional(T&& v) : OptionalBase(constexpr_move(v))
> {}
> [exec] ^
> [exec] 1 error generated.
> [exec] make[2]: ***
> [main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/configuration.cc.o]
> Error 1
> [exec] make[1]: ***
> [main/native/libhdfspp/lib/common/CMakeFiles/common_obj.dir/all] Error 2
> [exec] make: *** [all] Error 2
> {noformat}
> Compile error 2:
> {noformat}
> [exec]
> /Users/tiborkiss/workspace/apache-hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc:285:66:
> error: use of overloaded operator '<<' is ambiguous (with operand types
> 'hdfs::LogMessage' and 'size_type' (aka 'unsigned long'))
> [exec] << " Existing thread count = "
> << worker_threads_.size());
> [exec]
>
[jira] [Assigned] (HDFS-10339) libhdfs++: Expose async operations through the C API
[
https://issues.apache.org/jira/browse/HDFS-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer reassigned HDFS-10339:
--
Assignee: (was: James Clampffer)
> libhdfs++: Expose async operations through the C API
>
>
> Key: HDFS-10339
> URL: https://issues.apache.org/jira/browse/HDFS-10339
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Priority: Major
>
> I propose an API that looks like the following for doing async operations in
> C.
> (might be some typeos, going off memory of what I tried, will clean up)
> {code}
> typedef struct {
> int status;
> ssize_t count;
> ... whatever else ...
> } async_context;
> typedef void* caller_context;
> typedef void (*)(const async_context*, caller_context*) capi_callback;
> void hdfsAsyncPread(hdfsFS fs, hdfsFile file, off_t offset, void *buf, size_t
> count, capi_callback, caller_context);
> {code}
> When invoked we take a copy of the caller context that gets forwarded to the
> callback when the async op completes; this is where a user can keep a pointer
> to some state associated with the operation. The callback is invoked by a
> const async_contex* analogous to the Status object in the C++ API so the
> callback code can check status, bytes read, and other stuff.
> Internally this can be implemented by a callable struct/lambda that captures
> the caller_context and invokes the capi_callback with the caller_context and
> result async_context.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10339) libhdfs++: Expose async operations through the C API
[
https://issues.apache.org/jira/browse/HDFS-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10339:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: Expose async operations through the C API
>
>
> Key: HDFS-10339
> URL: https://issues.apache.org/jira/browse/HDFS-10339
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Priority: Major
>
> I propose an API that looks like the following for doing async operations in
> C.
> (might be some typeos, going off memory of what I tried, will clean up)
> {code}
> typedef struct {
> int status;
> ssize_t count;
> ... whatever else ...
> } async_context;
> typedef void* caller_context;
> typedef void (*)(const async_context*, caller_context*) capi_callback;
> void hdfsAsyncPread(hdfsFS fs, hdfsFile file, off_t offset, void *buf, size_t
> count, capi_callback, caller_context);
> {code}
> When invoked we take a copy of the caller context that gets forwarded to the
> callback when the async op completes; this is where a user can keep a pointer
> to some state associated with the operation. The callback is invoked by a
> const async_contex* analogous to the Status object in the C++ API so the
> callback code can check status, bytes read, and other stuff.
> Internally this can be implemented by a callable struct/lambda that captures
> the caller_context and invokes the capi_callback with the caller_context and
> result async_context.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10241) libhdfs++: Objects should never return mutable references to internal state
[
https://issues.apache.org/jira/browse/HDFS-10241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Clampffer updated HDFS-10241:
---
Parent Issue: HDFS-14032 (was: HDFS-8707)
> libhdfs++: Objects should never return mutable references to internal state
> ---
>
> Key: HDFS-10241
> URL: https://issues.apache.org/jira/browse/HDFS-10241
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Priority: Major
>
> Returning mutable references to internal state is always a bad idea. It's
> particularly bad in asynchronous code (due to unpredictable object life
> cycles).
> Example of what should _never_ happen in production code (from
> AsyncRequestBlock):
> {code}
> struct State {
> std::string header;
> hadoop::hdfs::OpReadBlockProto request;
> hadoop::hdfs::BlockOpResponseProto response;
> };
>
> auto m = continuation::Pipeline::Create(cancel_state_);
> State *s = >state();
>
> s->header.insert(s->header.begin(),
> {0, kDataTransferVersion, Operation::kReadBlock});
> {code}
> I'll open another JIRA for auto vars containing ambiguous types e.g. what is
> the type of "m" here?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9757) libhdfs++: examples fail to build with static libproto/openssl
[ https://issues.apache.org/jira/browse/HDFS-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9757: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: examples fail to build with static libproto/openssl > -- > > Key: HDFS-9757 > URL: https://issues.apache.org/jira/browse/HDFS-9757 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen >Priority: Major > Attachments: HDFS-9757.HDFS-8707.000.patch > > > If libproto is statically linked with C++, or libssl is statically linked by > requires libdl, the examples will fail to compile. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9758) libhdfs++: Implement Python bindings
[ https://issues.apache.org/jira/browse/HDFS-9758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9758: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: Implement Python bindings > > > Key: HDFS-9758 > URL: https://issues.apache.org/jira/browse/HDFS-9758 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Tibor Kiss >Priority: Major > Attachments: hdfs_posix.py > > > It'd be really useful to have bindings for various scripting languages. > Python would be a good start because of it's popularity and how easy it is to > interact with shared libraries using the ctypes module. I think bindings for > the V8 engine that nodeJS uses would be a close second in terms of expanding > the potential user base. > Probably worth starting with just adding a synchronous API and building from > there to avoid interactions with python's garbage collector until the > bindings prove to be solid. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9562) libhdfs++ RpcConnectionImpl::Connect should acquire connection state lock
[ https://issues.apache.org/jira/browse/HDFS-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9562: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++ RpcConnectionImpl::Connect should acquire connection state lock > - > > Key: HDFS-9562 > URL: https://issues.apache.org/jira/browse/HDFS-9562 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Priority: Critical > > RpcConnectionImpl::Connect calls pending_requests_.push_back() without > holding the connection_state_lock_. This isn't a huge issue at the moment > because Connect generally shouldn't be called on the same instance from many > threads but it wouldn't hurt to add to prevent future confusion. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9610) test_libhdfs_threaded_hdfs_static generates a lot of noise on stderr which looks like a failure even though it isn't
[ https://issues.apache.org/jira/browse/HDFS-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9610: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > test_libhdfs_threaded_hdfs_static generates a lot of noise on stderr which > looks like a failure even though it isn't > > > Key: HDFS-9610 > URL: https://issues.apache.org/jira/browse/HDFS-9610 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Allen Wittenauer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-9610.HDFS-8707.000.patch, LastTest.log > > > Playing around with adding ctest output support to Yetus, and I stumbled upon > a case where the tests throw errors left and right but claim success. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9539) libhdfs++: enable default configuration files
[ https://issues.apache.org/jira/browse/HDFS-9539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9539: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++: enable default configuration files > - > > Key: HDFS-9539 > URL: https://issues.apache.org/jira/browse/HDFS-9539 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen >Priority: Major > > In the Java implementation of config files, the Hadoop jars included a > default core-default and hdfs-default.xml file that provided default values > for the run-time configurations. libhdfs++ should honor that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9416) Respect OpenSSL and protobuf definitions in maven configuration when building libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-9416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9416: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Respect OpenSSL and protobuf definitions in maven configuration when building > libhdfspp > --- > > Key: HDFS-9416 > URL: https://issues.apache.org/jira/browse/HDFS-9416 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: Xiaobing Zhou >Priority: Blocker > Attachments: HDFS-9416.004.patch, HDFS-9416.HDFS-8707.004.patch, > HDFS-9416.HDFS-8707.005.patch > > > As discovered in HDFS-9380 the current pom.xml / CMakeLists.txt in libhdfspp > does not respect the configuration from the maven command line. Subsequently > it breaks the Jenkins build. > Both pom.xml and CMakeLists.txt need to be fixed to get Jenkins working again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9385) Enable config substitutions in libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-9385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9385: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Enable config substitutions in libhdfs++ > > > Key: HDFS-9385 > URL: https://issues.apache.org/jira/browse/HDFS-9385 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > Attachments: HDFS-9385.patch > > > The Java configuration code allows substitution of config values within the > config file and from the environment. The C++ code should be able to > correctly read any XML file meant to be parsed by the java code. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9328) Formalize coding standards for libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9328: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Formalize coding standards for libhdfs++ > > > Key: HDFS-9328 > URL: https://issues.apache.org/jira/browse/HDFS-9328 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Blocker > Fix For: HDFS-8707 > > Attachments: HDFS-9328.HDFS-8707.000.patch, > HDFS-9328.HDFS-8707.001.patch, HDFS-9328.HDFS-8707.002.patch, > HDFS-9328.HDFS-8707.003.patch, HDFS-9328.HDFS-8707.004.patch > > > We have 2-3 people working on this project full time and hopefully more > people will start contributing. In order to efficiently scale we need a > single, easy to find, place where developers can check to make sure they are > following the coding standards of this project to both save their time and > save the time of people doing code reviews. > The most practical place to do this seems like a README file in libhdfspp/. > The foundation of the standards is google's C++ guide found here: > https://google-styleguide.googlecode.com/svn/trunk/cppguide.html > Any exceptions to google's standards or additional restrictions need to be > explicitly enumerated so there is one single point of reference for all > libhdfs++ code standards. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9233) Create LICENSE.txt and NOTICES files for libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-9233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9233: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Create LICENSE.txt and NOTICES files for libhdfs++ > -- > > Key: HDFS-9233 > URL: https://issues.apache.org/jira/browse/HDFS-9233 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen >Priority: Major > > We use third-party libraries that are Apache and Google licensed, and may be > adding an MIT-licenced third-party library. We need to include the > appropriate license files for inclusion into Apache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9306) libhdfs++ configuration should do deprecated key lookup
[ https://issues.apache.org/jira/browse/HDFS-9306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9306: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > libhdfs++ configuration should do deprecated key lookup > --- > > Key: HDFS-9306 > URL: https://issues.apache.org/jira/browse/HDFS-9306 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Priority: Major > Fix For: HDFS-8707 > > > Several of the configuration keys for client settings can come from > deprecated settings; see DFSConfigKeys.java for anything deprecated to > HdfsClientConfigKeys.* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9227) Check block checksums for integrity
[ https://issues.apache.org/jira/browse/HDFS-9227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9227: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Check block checksums for integrity > --- > > Key: HDFS-9227 > URL: https://issues.apache.org/jira/browse/HDFS-9227 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-9227.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9115) Create documentation to describe the overall architecture and rationales of the library
[ https://issues.apache.org/jira/browse/HDFS-9115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9115: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Create documentation to describe the overall architecture and rationales of > the library > --- > > Key: HDFS-9115 > URL: https://issues.apache.org/jira/browse/HDFS-9115 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Haohui Mai >Assignee: James Clampffer >Priority: Major > Fix For: HDFS-8707 > > > It's beneficial to have documentations to describe the design decisions and > rationales of the library. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8790) Add Filesystem level stress tests
[ https://issues.apache.org/jira/browse/HDFS-8790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-8790: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Add Filesystem level stress tests > - > > Key: HDFS-8790 > URL: https://issues.apache.org/jira/browse/HDFS-8790 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu >Priority: Major > Attachments: HDFS-8790.HDFS-8707.000.patch, > HDFS-8790.HDFS-8707.001.patch, HDFS-8790.HDFS-8707.002.patch > > > I propose adding stress tests on libhdfs(3) compatibility layer was well as > the async calls. These can be also used for basic performance metrics and > inputs to profiling tools to see improvements over time. > I'd like to make these tests into a seperate executable, or set of them, so > that they can be used for longer running tests on dedicated clusters that may > already exist. Each should provide a simple command line interface for > scripted or manual use. > Basic tests would be: > looped open-read-close > sequential scans > small random reads > All tests will be parameterized for number of threads, read size, and upper > and lower offset bounds for a specified file. This will make it much easier > to detect and reproduce threading issues and resource leaks as well as > provide a simple executable (or set of executables) that can be run with > valgrind to gain a high confidence that the code is operating correctly. > I'd appreciate suggestions for any other simple stress tests. > HDFS-8766 intentionally avoided shared_ptr and unique_ptr in the C api to > make debugging this a little easier in case memory stomps and dangling > references show up in stress tests. These will be added into the C API when > the patch for this jira is submitted because things should be reasonably > stable once the stress tests pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8765) Implement local block reader in libhdfspp
[ https://issues.apache.org/jira/browse/HDFS-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-8765: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Implement local block reader in libhdfspp > - > > Key: HDFS-8765 > URL: https://issues.apache.org/jira/browse/HDFS-8765 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: short-circuit-demo.patch2, short-circuit-demo.patch3, > short-circuit-demo.patch4, short_circuit_demo.patch > > > Implement a block reader that uses the hdfs short circuit protocol to read > colocated data as efficiently as possible. Implementation will be based on > BlockReaderLocal.java + the associated JNI bindings. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8746) Reduce the latency of streaming reads by re-using DN connections
[ https://issues.apache.org/jira/browse/HDFS-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-8746: -- Parent Issue: HDFS-14032 (was: HDFS-8707) > Reduce the latency of streaming reads by re-using DN connections > > > Key: HDFS-8746 > URL: https://issues.apache.org/jira/browse/HDFS-8746 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: James Clampffer >Priority: Major > > The current libhdfspp implementation opens a new connection for each pread. > For streaming reads (especially streaming short-buffer reads coming from the > C API, and especially once we get SSL handshake overhead), our throughput > will be dominated by the connection latency of reconnecting to the DataNodes. > The target use case is a multi-block file that is being sequentially streamed > and processed by the client application, which consumes the data as it comes > from the DN and throws it away. The data is read into moderately small > buffers (~64k - ~1MB) owned by the consumer, and overall throughput is the > critical metric. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-8707) Implement an async pure c++ HDFS client
[ https://issues.apache.org/jira/browse/HDFS-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer resolved HDFS-8707. --- Resolution: Fixed Marking resolved since this has been merged to trunk. Moving remaining work under HDFS-14032. > Implement an async pure c++ HDFS client > --- > > Key: HDFS-8707 > URL: https://issues.apache.org/jira/browse/HDFS-8707 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Reporter: Owen O'Malley >Assignee: James Clampffer >Priority: Major > Fix For: 3.2.0 > > > As part of working on the C++ ORC reader at ORC-3, we need an HDFS pure C++ > client that lets us do async io to HDFS. We want to start from the code that > Haohui's been working on at https://github.com/haohui/libhdfspp . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14032) [libhdfs++] Phase 2 improvements
James Clampffer created HDFS-14032: -- Summary: [libhdfs++] Phase 2 improvements Key: HDFS-14032 URL: https://issues.apache.org/jira/browse/HDFS-14032 Project: Hadoop HDFS Issue Type: Task Components: hdfs-client Reporter: James Clampffer Assignee: James Clampffer HDFS-8707 (libhdfs++) was merged to trunk, this is an umbrella JIRA for things that still need to get done. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8707) Implement an async pure c++ HDFS client
[ https://issues.apache.org/jira/browse/HDFS-8707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665112#comment-16665112 ] James Clampffer commented on HDFS-8707: --- [~sunilg] Sure, I'll resolve this and make a new umbrella Jira and move subtasks that weren't completed prior to the merge into that. I hadn't realized this was causing confusion for managing releases, though that makes sense. Sorry about that! Just to make sure I understand which Fixed version to use: should all subtasks that were committed on the HDFS-8707 branch prior to merging to trunk be marked with the same fix version as HDFS-8707 since that's when they were resolved on trunk? > Implement an async pure c++ HDFS client > --- > > Key: HDFS-8707 > URL: https://issues.apache.org/jira/browse/HDFS-8707 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client >Reporter: Owen O'Malley >Assignee: James Clampffer >Priority: Major > Fix For: 3.2.0 > > > As part of working on the C++ ORC reader at ORC-3, we need an HDFS pure C++ > client that lets us do async io to HDFS. We want to start from the code that > Haohui's been working on at https://github.com/haohui/libhdfspp . -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[
https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664057#comment-16664057
]
James Clampffer commented on HDFS-12026:
{quote}I think there are no progress. To unblock this compatibility issue,
could this be reverted?
{quote}
[~sunilg] The HDFS-8707 branch got voted in, and plenty of patches have been
applied on top of that on trunk since. Pulling out an entire component that
people use over build issues that did not exist when it was merged to trunk
last July doesn't seem like an appropriate solution. Patching the code to deal
with platform support changes is less work and keeps the functionality in
place. I can help with that if I know what needs to be done.
Can you show me what build errors you're getting or give me some more details
about what exactly needs to be supported? It'd be easier to track if you filed
a new JIRA since this patch was only intended to deal with different default
warnings between clang and GCC.
> libhdfs++: Fix compilation errors and warnings when compiling with Clang
> -
>
> Key: HDFS-12026
> URL: https://issues.apache.org/jira/browse/HDFS-12026
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
>Priority: Blocker
> Attachments: HDFS-12026.HDFS-8707.000.patch,
> HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch,
> HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch,
> HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch,
> HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch,
> HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch
>
>
> Currently multiple errors and warnings prevent libhdfspp from being compiled
> with clang. It should compile cleanly using flag:
> -std=c++11
> and also warning flags:
> -Weverything -Wno-c++98-compat -Wno-missing-prototypes
> -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default
> -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[ https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660597#comment-16660597 ] James Clampffer commented on HDFS-11520: [~anatoli.shein] I'm going to start reviewing today. Might take a couple days to get through since it's a large and complex patch. > libhdfs++: Support cancellation of individual RPC calls in C++ API > -- > > Key: HDFS-11520 > URL: https://issues.apache.org/jira/browse/HDFS-11520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch, > HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.007.patch, > HDFS-11520.008.patch, HDFS-11520.009.patch, HDFS-11520.010.patch, > HDFS-11520.011.patch, HDFS-11520.012.patch, HDFS-11520.013.patch, > HDFS-11520.014.patch, HDFS-11520.015.patch, HDFS-11520.016.patch, > HDFS-11520.017.patch, HDFS-11520.018.patch, HDFS-11520.019.patch, > HDFS-11520.HDFS-8707.000.patch, HDFS-11520.trunk.001.patch > > > RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be > individually cancelable without impacting other pending RPC calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13745) libhdfs++: Fix race in FileSystem destructor
[
https://issues.apache.org/jira/browse/HDFS-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590702#comment-16590702
]
James Clampffer commented on HDFS-13745:
Thanks for checkout this out [~anatoli.shein].
{quote}Is there a possibility that some task executed by the IoService will run
forever?
{quote}
Yes. Someone can pass in a callback that does a long sleep or busy wait.
There's plenty of comments that say you should never pass in a callback that
can block for an indeterminate amount of time; there's nothing the library can
do if someone chooses to ignore those. All of the internal tasks that the
library runs in the ioservice context have timeouts to prevent them from
running forever.
{quote}Should we add some timeout in BlockingStop method if we have been
waiting too long?
{quote}
No. BlockingStop is only there to prevent a thread self-join (and only blocks
if that's what would happen otherwise). The only thing exiting the loop early
can do is let the self join happen. On the surface it looks like you could
spawn another thread and run the dtor there but then you're stick with a
similar issue when it comes to managing the lifetime of that thread.
{quote}In the hdfs_ioservice_test in longRunningCallback we sleep for just 1
second, which might not be enough since if there is some sort of system delay
longer than 1 second the test might fail. Even though with any amount of sleep
there is a chance of this happening, it might make sense to increase it to 2-3
seconds.
{quote}
Increasing the sleep to 2 or 3 seconds seems just as arbitrary as a 1 second
sleep. I'll see if I can get rid of the sleep by adding an extra condition
variable.
bq. Also, can we submit another CI run for this? Looks like the previous one
didn't run for some reason.
Yeah. I'll do that once I add the condition variable.
> libhdfs++: Fix race in FileSystem destructor
>
>
> Key: HDFS-13745
> URL: https://issues.apache.org/jira/browse/HDFS-13745
> Project: Hadoop HDFS
> Issue Type: Task
> Components: native
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
> Attachments: HDFS-13745.000.patch
>
>
> Whatever happens to have the last shared_ptr to the IoService will run
> ~IoService when the shared_ptr goes out of scope. IoService's destructor is
> responsible for joining all worker threads in the pool. Most callbacks now
> own weak_ptr that can be promoted to a shared_ptr in order to post
> new async tasks. If a callback object is the last thing holding the
> IoService shared_ptr it's going to try to join the thread pool inside of one
> of the thread pool's threads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13822) speedup libhdfs++ build (enable parallel build)
[ https://issues.apache.org/jira/browse/HDFS-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583025#comment-16583025 ] James Clampffer commented on HDFS-13822: [~jlowe] I agree that this isn't breaking the tests any more than they already were. One of the native client issues is libhdfs++ related (HDFS-9610), however it looks like that provided cover for another bug to be introduced in the JNI libhdfs hdfsGetLastExceptionRootCause a month or two ago. The only test coverage added for hdfsGetLastExceptionRootCause was one call in test_libhdfs_threaded (both libhdfs and libhdfs++ run that test). Now it deterministically returns a null String ref rather than the expected output. I noticed that when I was trying to sort out HDFS-9610 and add test coverage for the libhdfs++ C++ API but haven't had a chance to figure out which commit broke it. > speedup libhdfs++ build (enable parallel build) > --- > > Key: HDFS-13822 > URL: https://issues.apache.org/jira/browse/HDFS-13822 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Pradeep Ambati >Assignee: Allen Wittenauer >Priority: Minor > Attachments: HDFS-13382.000.patch, HDFS-13822.01.patch, > HDFS-13822.02.patch > > > libhdfs++ has significantly increased clean build times for the native client > on trunk. Problem is that libhdfs++ isn't build in parallel. When I tried to > force a parallel build by specifying -Dnative_make_args=-j4, the build fails > due to dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13745) libhdfs++: Fix race in FileSystem destructor
[ https://issues.apache.org/jira/browse/HDFS-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548123#comment-16548123 ] James Clampffer commented on HDFS-13745: Patch ready for review. asio's io_service::stop() only prevents future work from being run and doesn't halt callbacks that are running. I added a BlockingStop method to the hdfs::IoService that keeps a count of how many tasks are being executed by the IoService and blocks until that's zero. The FileSystem destructor now uses BlockingStop to make sure tasks are gone before the IoService is destroyed. > libhdfs++: Fix race in FileSystem destructor > > > Key: HDFS-13745 > URL: https://issues.apache.org/jira/browse/HDFS-13745 > Project: Hadoop HDFS > Issue Type: Task > Components: native >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13745.000.patch > > > Whatever happens to have the last shared_ptr to the IoService will run > ~IoService when the shared_ptr goes out of scope. IoService's destructor is > responsible for joining all worker threads in the pool. Most callbacks now > own weak_ptr that can be promoted to a shared_ptr in order to post > new async tasks. If a callback object is the last thing holding the > IoService shared_ptr it's going to try to join the thread pool inside of one > of the thread pool's threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13745) libhdfs++: Fix race in FileSystem destructor
[ https://issues.apache.org/jira/browse/HDFS-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13745: --- Status: Patch Available (was: Open) > libhdfs++: Fix race in FileSystem destructor > > > Key: HDFS-13745 > URL: https://issues.apache.org/jira/browse/HDFS-13745 > Project: Hadoop HDFS > Issue Type: Task > Components: native >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13745.000.patch > > > Whatever happens to have the last shared_ptr to the IoService will run > ~IoService when the shared_ptr goes out of scope. IoService's destructor is > responsible for joining all worker threads in the pool. Most callbacks now > own weak_ptr that can be promoted to a shared_ptr in order to post > new async tasks. If a callback object is the last thing holding the > IoService shared_ptr it's going to try to join the thread pool inside of one > of the thread pool's threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13745) libhdfs++: Fix race in FileSystem destructor
[ https://issues.apache.org/jira/browse/HDFS-13745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13745: --- Attachment: HDFS-13745.000.patch > libhdfs++: Fix race in FileSystem destructor > > > Key: HDFS-13745 > URL: https://issues.apache.org/jira/browse/HDFS-13745 > Project: Hadoop HDFS > Issue Type: Task > Components: native >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13745.000.patch > > > Whatever happens to have the last shared_ptr to the IoService will run > ~IoService when the shared_ptr goes out of scope. IoService's destructor is > responsible for joining all worker threads in the pool. Most callbacks now > own weak_ptr that can be promoted to a shared_ptr in order to post > new async tasks. If a callback object is the last thing holding the > IoService shared_ptr it's going to try to join the thread pool inside of one > of the thread pool's threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13745) libhdfs++: Fix race in FileSystem destructor
James Clampffer created HDFS-13745: -- Summary: libhdfs++: Fix race in FileSystem destructor Key: HDFS-13745 URL: https://issues.apache.org/jira/browse/HDFS-13745 Project: Hadoop HDFS Issue Type: Task Components: native Reporter: James Clampffer Assignee: James Clampffer Whatever happens to have the last shared_ptr to the IoService will run ~IoService when the shared_ptr goes out of scope. IoService's destructor is responsible for joining all worker threads in the pool. Most callbacks now own weak_ptr that can be promoted to a shared_ptr in order to post new async tasks. If a callback object is the last thing holding the IoService shared_ptr it's going to try to join the thread pool inside of one of the thread pool's threads. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11520) libhdfs++: Support cancellation of individual RPC calls in C++ API
[
https://issues.apache.org/jira/browse/HDFS-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16515968#comment-16515968
]
James Clampffer commented on HDFS-11520:
Thanks for picking this up [~anatoli.shein]!
It doesn't look like you're touching the java build so I doubt the shadedclient
failures are related to this. Shading just does a name mangling pass on java
class files to avoid library versioning issues.
Some general feedback:
-Some of the comments in my original patch are no longer applicable (I had
never intended to merge in that state). If you're basing this work off of that
make sure to update or remove them where appropriate. There's several in
rpc_connection.h that were reminders for future work I want to do but aren't
directly related to this. Example where the comment is alluding a change that
already happened so it's just confusing now:
{code:java}
+ // This should be an hdfs::IoService.
+ // Can get away for it at the moment because the FileSystemImpl is the only
thing that uses it.
std::shared_ptr io_service_;
{code}
-Certain RPC (mainly the recursive ones) can't be cancelled right now and just
no-op. Rather than silently no-op explicitly fail and log a warning where it's
invoked so the user is aware of this.
-Break the unit test up into multiple blocks. StatCancel is also testing a lot
of other RPC. I'd most likely write a macro or templated function that allows
a method name and args to be plugged in for different calls rather than
repeating the same 15ish lines for each call but that's mostly preference.
-Add unit tests that do cancel in other phases of RPC. Right now the only place
being tested is when the NN response is being decoded. Realistically cancel is
going to be used when the network is stalled or NN is doing GC so you want to
test cancel of outgoing messages, messages in the queue to be sent, messages
that have been sent but a response hasn't been received etc.
-Add cancel tests to the mini stress tests. Cancelling from another thread
introduces potential for race conditions that are hard to perfectly emulate in
unit tests.
-Is this able to cancel requests that have shifted to a new RPCConnection due
to failover? What about retry?
> libhdfs++: Support cancellation of individual RPC calls in C++ API
> --
>
> Key: HDFS-11520
> URL: https://issues.apache.org/jira/browse/HDFS-11520
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: Anatoli Shein
>Priority: Major
> Attachments: HDFS-11520.002.patch, HDFS-11520.003.patch,
> HDFS-11520.004.patch, HDFS-11520.005.patch, HDFS-11520.HDFS-8707.000.patch,
> HDFS-11520.trunk.001.patch
>
>
> RPC calls done by FileSystem methods like Mkdirs, GetFileInfo etc should be
> individually cancelable without impacting other pending RPC calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13535) Fix libhdfs++ doxygen build
[ https://issues.apache.org/jira/browse/HDFS-13535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508069#comment-16508069 ] James Clampffer commented on HDFS-13535: [~mtracy], thanks for doing this! Some feedback: -Is it possible to suppress warnings about undocumented classes without annotating the code? I get a lot of those when building the doc. -The dockerfile in dev-support/docker doesn't include graphviz so that generates errors. We may just want to suppress those warnings for now as well. -Before we invest too much time in doxygen infrastructure we should look into the concerns [~aw] raised in HDFS-8745. That said getting the current doc build to work is much better than having no doc build. > Fix libhdfs++ doxygen build > --- > > Key: HDFS-13535 > URL: https://issues.apache.org/jira/browse/HDFS-13535 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.2 >Reporter: Mitchell Tracy >Assignee: Mitchell Tracy >Priority: Major > Attachments: HDFS-13535.0.patch > > > Currently, the doxygen build for libhdfs++ doesn't include all of the > necessary source directories. In addition, the build does not generate the > actual html documentation. So the fix is to include all the required source > directories when generating the doxyfile, and then add maven for generating > the html documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13666) Only use existing SASL libraries in libhdfs++
[ https://issues.apache.org/jira/browse/HDFS-13666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13666: --- Description: At one point there were plans to implement SASL support from scratch inside of libhdfs+\+. Since then we added added Kerberos support through Cyrus SASL and GSasl. I propose removing the partial implementation from libhdfs++ since doing the secure handshake and encryption are still marked TODO but it's unclear what getting that done involves. That way we can focus time on using Cyrus (GSasl uses GPL) to support DIGEST-MD5 auth and wire encryption. (was: At one point there were plans to implement SASL support from scratch inside of libhdfs++. Since then we added added Kerberos support through Cyrus SASL and GSasl. I propose removing the partial implementation from libhdfs++ since doing the secure handshake and encryption are still marked TODO but it's unclear what getting that done involves. That way we can focus time on using Cyrus (GSasl uses GPL) to support DIGEST-MD5 auth and wire encryption.) > Only use existing SASL libraries in libhdfs++ > - > > Key: HDFS-13666 > URL: https://issues.apache.org/jira/browse/HDFS-13666 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Priority: Major > > At one point there were plans to implement SASL support from scratch inside > of libhdfs+\+. Since then we added added Kerberos support through Cyrus SASL > and GSasl. I propose removing the partial implementation from libhdfs++ since > doing the secure handshake and encryption are still marked TODO but it's > unclear what getting that done involves. That way we can focus time on using > Cyrus (GSasl uses GPL) to support DIGEST-MD5 auth and wire encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13666) Only use existing SASL libraries in libhdfs++
James Clampffer created HDFS-13666: -- Summary: Only use existing SASL libraries in libhdfs++ Key: HDFS-13666 URL: https://issues.apache.org/jira/browse/HDFS-13666 Project: Hadoop HDFS Issue Type: Task Reporter: James Clampffer At one point there were plans to implement SASL support from scratch inside of libhdfs++. Since then we added added Kerberos support through Cyrus SASL and GSasl. I propose removing the partial implementation from libhdfs++ since doing the secure handshake and encryption are still marked TODO but it's unclear what getting that done involves. That way we can focus time on using Cyrus (GSasl uses GPL) to support DIGEST-MD5 auth and wire encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13615) libhdfs++ SaslProtocol hanging while accessing invalid lock
[ https://issues.apache.org/jira/browse/HDFS-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13615: --- Resolution: Fixed Status: Resolved (was: Patch Available) I committed this to trunk. Thanks for your contribution [~mtracy]! > libhdfs++ SaslProtocol hanging while accessing invalid lock > --- > > Key: HDFS-13615 > URL: https://issues.apache.org/jira/browse/HDFS-13615 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Mitchell Tracy >Assignee: Mitchell Tracy >Priority: Major > Attachments: HDFS-13615.000.patch > > > During certain race conditions, it is possible to have an asio thread that > wakes up to process a response, but the RpcConnection to which this > SaslProtocol belongs is dead. > This can happen as a result of the filesystem being destroyed in the middle > of a sasl handshake. > Therefore, we should explicitly capture the RpcConnection in the lambda to > ensure its existence when the SaslProtocol wakes up to process the server's > response. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13534: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for reviewing [~owen.omalley]! I'll get a PR up on the ORC side shortly. > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch, HDFS-13534.001.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13615) libhdfs++ SaslProtocol hanging while accessing invalid lock
[ https://issues.apache.org/jira/browse/HDFS-13615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503960#comment-16503960 ] James Clampffer commented on HDFS-13615: [~mtracy] Nice! We've had to do similar things elsewhere due to lambdas not capturing things they depend on. +1. I'll commit shortly. This patch can end up causing an IoService thread to attempt to join with itself: the lambda props up the lifetime of the SaslProtocol, which holds a shared_ptr to the RpcEngine which in turn holds a shared_ptr to the IoService. If the lambda is the last thing holding the refcount > 0 then those destructors get run in an ioservice thread, and the ioservice destructor tries to join all ioservice threads. Your fix is still an improvement as the destruction order issues had already been around. Those can be addressed in another jira. > libhdfs++ SaslProtocol hanging while accessing invalid lock > --- > > Key: HDFS-13615 > URL: https://issues.apache.org/jira/browse/HDFS-13615 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Mitchell Tracy >Assignee: Mitchell Tracy >Priority: Major > Attachments: HDFS-13615.000.patch > > > During certain race conditions, it is possible to have an asio thread that > wakes up to process a response, but the RpcConnection to which this > SaslProtocol belongs is dead. > This can happen as a result of the filesystem being destroyed in the middle > of a sasl handshake. > Therefore, we should explicitly capture the RpcConnection in the lambda to > ensure its existence when the SaslProtocol wakes up to process the server's > response. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503849#comment-16503849 ] James Clampffer commented on HDFS-13534: [~owen.omalley] I think that's because the version in ORC is fairly old. I've cleaned up a lot of stuff since then, mostly related to memory management but also getting rid of extra stuff in headers, so the patch isn't going to be the same. Want me to add a PR to update the libhdfs++ version in ORC? > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch, HDFS-13534.001.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468007#comment-16468007 ] James Clampffer commented on HDFS-13534: Thanks for looking at this [~anatoli.shein]. I attached a new patch. The npm stuff wasn't supposed to be part of this; I comment that out on my working tree because container builds always seem to hang there for some reason. I'm not able to reproduce the override warning you're seeing, what compiler/version are you using? Right now I think the important part is to get functional included since that breaks the build. The warnings fixed here are the ones I was able to hit when using clang in the docker container. > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch, HDFS-13534.001.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13534: --- Attachment: HDFS-13534.001.patch > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch, HDFS-13534.001.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13429) libhdfs++ Expose a C++ logging API
[ https://issues.apache.org/jira/browse/HDFS-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467532#comment-16467532 ] James Clampffer commented on HDFS-13429: New patch up without trailing whitespace. Tests are included. C++ test updates don't show up in test4tests. > libhdfs++ Expose a C++ logging API > -- > > Key: HDFS-13429 > URL: https://issues.apache.org/jira/browse/HDFS-13429 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13429.000.patch, HDFS-13429.001.patch > > > The libhdfs++ C API supports taking function pointers for log plugins and > defines levels for components and severity. It'd be nice to have an > idiomatic C++ logging interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13429) libhdfs++ Expose a C++ logging API
[ https://issues.apache.org/jira/browse/HDFS-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13429: --- Attachment: HDFS-13429.001.patch > libhdfs++ Expose a C++ logging API > -- > > Key: HDFS-13429 > URL: https://issues.apache.org/jira/browse/HDFS-13429 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13429.000.patch, HDFS-13429.001.patch > > > The libhdfs++ C API supports taking function pointers for log plugins and > defines levels for components and severity. It'd be nice to have an > idiomatic C++ logging interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13403) libhdfs++: Use hdfs::IoService object rather than asio::io_service
[ https://issues.apache.org/jira/browse/HDFS-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467494#comment-16467494 ] James Clampffer commented on HDFS-13403: Thanks for finding that [~bibinchundatt]! [~pifta] I posted a patch on HDFS-13534 to fix this if you'd like to review. > libhdfs++: Use hdfs::IoService object rather than asio::io_service > -- > > Key: HDFS-13403 > URL: https://issues.apache.org/jira/browse/HDFS-13403 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13403.000.patch, build_fixes.patch > > > At the moment the hdfs::IoService is a simple wrapper over asio's io_service > object. I'd like to make this smarter and have it do things like track which > tasks are queued, validate that dependencies of tasks exist, and monitor > ioservice throughput and contention. In order to get there we need to use > have all components in the library to go through the hdfs::IoService rather > than directly interacting with the asio::io_service. The only time the > asio::io_service should be used is when calling things like asio::async_write > that need an io_service&. HDFS-11884 will be able get rid of those remaining > instances once this work is in place. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13534: --- Attachment: HDFS-13534.000.patch > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13534) libhdfs++: Fix GCC7 build
[ https://issues.apache.org/jira/browse/HDFS-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13534: --- Status: Patch Available (was: Open) > libhdfs++: Fix GCC7 build > - > > Key: HDFS-13534 > URL: https://issues.apache.org/jira/browse/HDFS-13534 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-13534.000.patch > > > After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. > [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex > implicitly included functional. Without that implicit include the compiler > errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13534) libhdfs++: Fix GCC7 build
James Clampffer created HDFS-13534: -- Summary: libhdfs++: Fix GCC7 build Key: HDFS-13534 URL: https://issues.apache.org/jira/browse/HDFS-13534 Project: Hadoop HDFS Issue Type: Task Reporter: James Clampffer Assignee: James Clampffer After merging HDFS-13403 [~pifta] noticed the build broke on some platforms. [~bibinchundatt] pointed out that prior to gcc 7 mutex, future, and regex implicitly included functional. Without that implicit include the compiler errors on the std::function in ioservice.h. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13429) libhdfs++ Expose a C++ logging API
[ https://issues.apache.org/jira/browse/HDFS-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13429: --- Status: Patch Available (was: Open) > libhdfs++ Expose a C++ logging API > -- > > Key: HDFS-13429 > URL: https://issues.apache.org/jira/browse/HDFS-13429 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13429.000.patch > > > The libhdfs++ C API supports taking function pointers for log plugins and > defines levels for components and severity. It'd be nice to have an > idiomatic C++ logging interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13429) libhdfs++ Expose a C++ logging API
[ https://issues.apache.org/jira/browse/HDFS-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461399#comment-16461399 ] James Clampffer commented on HDFS-13429: First patch ready for review. Comments: -The logger was already done in C++ but the interface didn't make it into the include/hdfspp directory. Most of this patch is just making the existing code consistent so it could be part of the public API. -C logging API that was in hdfspp/log.h was moved into hdfspp/hdfs_ext.h with the rest of the C API. This was done to avoid confusion over which headers were pure C and which used C++. The C++ logging API now lives in hdfspp/log.h. -Added a new test to make sure constants exposed in the public API aren't accidentally changed. > libhdfs++ Expose a C++ logging API > -- > > Key: HDFS-13429 > URL: https://issues.apache.org/jira/browse/HDFS-13429 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13429.000.patch > > > The libhdfs++ C API supports taking function pointers for log plugins and > defines levels for components and severity. It'd be nice to have an > idiomatic C++ logging interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13429) libhdfs++ Expose a C++ logging API
[ https://issues.apache.org/jira/browse/HDFS-13429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13429: --- Attachment: HDFS-13429.000.patch > libhdfs++ Expose a C++ logging API > -- > > Key: HDFS-13429 > URL: https://issues.apache.org/jira/browse/HDFS-13429 > Project: Hadoop HDFS > Issue Type: Task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13429.000.patch > > > The libhdfs++ C API supports taking function pointers for log plugins and > defines levels for components and severity. It'd be nice to have an > idiomatic C++ logging interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-11807: --- Resolution: Fixed Status: Resolved (was: Patch Available) > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch, > HDFS-11807.HDFS-8707.007.patch, HDFS-11807.HDFS-8707.008.patch, > HDFS-11807.HDFS-8707.009.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11807) libhdfs++: Get minidfscluster tests running under valgrind
[ https://issues.apache.org/jira/browse/HDFS-11807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461217#comment-16461217 ] James Clampffer commented on HDFS-11807: +1, I just committed this to trunk. Sorry it took so long to get around to it. Some tests I did: -Build with gcc and clang in the dev-support docker container and ran tests -Add memory leaks to make sure it caught them -Add invalid reads and writes to make sure they were caught > libhdfs++: Get minidfscluster tests running under valgrind > -- > > Key: HDFS-11807 > URL: https://issues.apache.org/jira/browse/HDFS-11807 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11807.HDFS-8707.000.patch, > HDFS-11807.HDFS-8707.001.patch, HDFS-11807.HDFS-8707.002.patch, > HDFS-11807.HDFS-8707.003.patch, HDFS-11807.HDFS-8707.004.patch, > HDFS-11807.HDFS-8707.005.patch, HDFS-11807.HDFS-8707.006.patch, > HDFS-11807.HDFS-8707.007.patch, HDFS-11807.HDFS-8707.008.patch, > HDFS-11807.HDFS-8707.009.patch > > > The gmock based unit tests generally don't expose race conditions and memory > stomps. A good way to expose these is running libhdfs++ stress tests and > tools under valgrind and pointing them at a real cluster. Right now the CI > tools don't do that so bugs occasionally slip in and aren't caught until they > cause trouble in applications that use libhdfs++ for HDFS access. > The reason the minidfscluster tests don't run under valgrind is because the > GC and JIT compiler in the embedded JVM do things that look like errors to > valgrind. I'd like to have these tests do some basic setup and then fork > into two processes: one for the minidfscluster stuff and one for the > libhdfs++ client test. A small amount of shared memory can be used to > provide a place for the minidfscluster to stick the hdfsBuilder object that > the client needs to get info about which port to connect to. Can also stick > a condition variable there to let the minidfscluster know when it can shut > down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13403) libhdfs++: Use hdfs::IoService object rather than asio::io_service
[ https://issues.apache.org/jira/browse/HDFS-13403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-13403: --- Attachment: build_fixes.patch > libhdfs++: Use hdfs::IoService object rather than asio::io_service > -- > > Key: HDFS-13403 > URL: https://issues.apache.org/jira/browse/HDFS-13403 > Project: Hadoop HDFS > Issue Type: Task >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-13403.000.patch, build_fixes.patch > > > At the moment the hdfs::IoService is a simple wrapper over asio's io_service > object. I'd like to make this smarter and have it do things like track which > tasks are queued, validate that dependencies of tasks exist, and monitor > ioservice throughput and contention. In order to get there we need to use > have all components in the library to go through the hdfs::IoService rather > than directly interacting with the asio::io_service. The only time the > asio::io_service should be used is when calling things like asio::async_write > that need an io_service&. HDFS-11884 will be able get rid of those remaining > instances once this work is in place. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
