[jira] [Commented] (ARROW-8587) [C++] Compilation error when linking arrow-flight-perf-server
[
https://issues.apache.org/jira/browse/ARROW-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135158#comment-17135158
]
Chengxin Ma commented on ARROW-8587:
[~apitrou] The problem is gone.
I checked with {{HEAD}} pointing to {{3ccecfb36}}.
{code}
Singularity> git rev-parse --short HEAD
3ccecfb36
{code}
> [C++] Compilation error when linking arrow-flight-perf-server
> -
>
> Key: ARROW-8587
> URL: https://issues.apache.org/jira/browse/ARROW-8587
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.17.0
> Environment: Linux HP 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar
> 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Chengxin Ma
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I wanted to play around with Flight benchmark after seeing the discussion
> regarding Flight's throughput in arrow dev mailing list today.
> I met the following error when trying to build the benchmark from latest
> source code:
> {code:java}
> [ 95%] Linking CXX executable ../../../debug/arrow-flight-perf-server
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::canonical(boost::filesystem::path const&,
> boost::filesystem::path const&, boost::system::error_code*)'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::system_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::parent_path() const'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::generic_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::current_path(boost::system::error_code*)'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `inflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `deflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::operator/=(boost::filesystem::path const&)'
> collect2: error: ld returned 1 exit status
> src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:154:
> recipe for target 'debug/arrow-flight-perf-server' failed
> make[2]: *** [debug/arrow-flight-perf-server] Error 1
> CMakeFiles/Makefile2:2609: recipe for target
> 'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
> make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
> Error 2
> Makefile:140: recipe for target 'all' failed
> make: *** [all] Error 2
> {code}
> I was using {{cmake .. -DCMAKE_BUILD_TYPE=Debug
> -DARROW_DEPENDENCY_SOURCE=AUTO -DARROW_FLIGHT=ON -DARROW_BUILD_BENCHMARKS=ON
> -DARROW_CXXFLAGS="-lboost_filesystem -lboost_system"}} to configure the build.
> I noticed that there was a {{ARROW_BOOST_BUILD_VERSION: 1.71.0}} in the
> output, but the Boost library that I installed from the package manger was of
> this version: {{1.65.1.0ubuntu1}}. Could this be the cause of the problem?
> PS:
> I was able to build the benchmark
> [before|https://issues.apache.org/jira/browse/ARROW-7200]. It was on AWS with
> the OS being ubuntu-bionic-18.04-amd64-server-20191002, which should be very
> similar to the one I'm using on my laptop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-8861) Memory not released until Plasma process is killed
Chengxin Ma created ARROW-8861:
--
Summary: Memory not released until Plasma process is killed
Key: ARROW-8861
URL: https://issues.apache.org/jira/browse/ARROW-8861
Project: Apache Arrow
Issue Type: Bug
Components: C++ - Plasma
Affects Versions: 0.16.0
Environment: Singularity container (Ubuntu 18.04)
Reporter: Chengxin Ma
Invoking the {{Delete(const ObjectID& object_id)}} method of a plasma client
seems not really to free up the memory used by the object.
To reproduce:
1. use {{htop}} (or other similar tools) to monitor memory usage;
2. start up the Plasma Object Store by {{plasma_store -m 10 -s
/tmp/plasma}};
3. use {{put.py}} to put an object into Plasma;
4. compile and run {{delete.cc}} ({{g++ delete.cc `pkg-config --cflags --libs
arrow plasma` --std=c++11 -o delete}});
5. kill the {{plasma_store}} process.
Memory usage drops at Step 5, rather than Step 4.
How to free up the memory while keeping Plasma Object Store running?
{{put.py}}:
{code:java}
from pyarrow import plasma
if __name__ == "__main__":
client = plasma.connect("/tmp/plasma")
object_id = plasma.ObjectID(20 * b"a")
object_size = 5
buffer = memoryview(client.create(object_id, object_size))
for i in range(5):
buffer[i] = i % 128
client.seal(object_id)
client.disconnect()
{code}
{{delete.cc}}:
{code:java}
#include "arrow/util/logging.h"
#include
using namespace plasma;
int main(int argc, char **argv)
{
PlasmaClient client;
ARROW_CHECK_OK(client.Connect("/tmp/plasma"));
ObjectID object_id = ObjectID::from_binary("");
client.Delete(object_id);
ARROW_CHECK_OK(client.Disconnect());
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-8587) Compilation error when linking arrow-flight-perf-server
[
https://issues.apache.org/jira/browse/ARROW-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092106#comment-17092106
]
Chengxin Ma commented on ARROW-8587:
I've built the Flight benchmark on different platforms before and never needed
to use {{ARROW_WITH_ZLIB=ON}}. I have no idea why it became necessary this time.
I just reproduced the compilation error in a Singularity container. The
container was built from [this definition
file|https://github.com/MaChengxin/ArrowSAM/blob/73aaf579c14929e78dcbc22ff54436ff17784901/Singularity/Singularity].
{code:java}
Singularity> cd temp/arrow/cpp/
Singularity> mkdir release
Singularity> cd release/
Singularity> cmake .. -DCMAKE_BUILD_TYPE=Release -DARROW_DEPENDENCY_SOURCE=AUTO
-DARROW_FLIGHT=ON -DARROW_BUILD_BENCHMARKS=ON
...
Singularity> make
...
../../../release/libarrow_flight.so.18.0.0: undefined reference to
`inflateInit2_'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `inflate'
../../../release/libarrow_flight.so.18.0.0: undefined reference to
`deflateInit2_'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `deflate'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
collect2: error: ld returned 1 exit status
src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:156: recipe
for target 'release/arrow-flight-perf-server' failed
make[2]: *** [release/arrow-flight-perf-server] Error 1
CMakeFiles/Makefile2:2648: recipe for target
'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
{code}
I cloned the latest apache/arrow repo (to temp/arrow) to ensure this problem is
not caused by my accidental mess-up to my local arrow repo.
The latest commit of the temporary clone is:
{code:java}
Singularity> git log
commit 045fe1382349485e6eb2ce9466757efedf207c94 (HEAD -> master, origin/master,
origin/HEAD)
{code}
> Compilation error when linking arrow-flight-perf-server
> ---
>
> Key: ARROW-8587
> URL: https://issues.apache.org/jira/browse/ARROW-8587
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.17.0
> Environment: Linux HP 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar
> 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Chengxin Ma
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I wanted to play around with Flight benchmark after seeing the discussion
> regarding Flight's throughput in arrow dev mailing list today.
> I met the following error when trying to build the benchmark from latest
> source code:
> {code:java}
> [ 95%] Linking CXX executable ../../../debug/arrow-flight-perf-server
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::canonical(boost::filesystem::path const&,
> boost::filesystem::path const&, boost::system::error_code*)'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::system_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::parent_path() const'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::generic_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::current_path(boost::system::error_code*)'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `inflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `deflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::operator/=(boost::filesystem::path const&)'
> collect2: error: ld returned 1 exit status
> src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:154:
> recipe for target 'debug/arrow-flight-perf-server' failed
> make[2]: *** [debug/arrow-flight-perf-server] Error 1
> CMakeFiles/Makefile2:2609: recipe for target
> 'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
> make[1]: ***
[jira] [Commented] (ARROW-8587) Compilation error when linking arrow-flight-perf-server
[
https://issues.apache.org/jira/browse/ARROW-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091885#comment-17091885
]
Chengxin Ma commented on ARROW-8587:
Adding {{-DARROW_WITH_ZLIB=ON}} solved this problem.
(I was expecting that the build system could find zlib on my system
automatically so I didn't set this flag.)
> Compilation error when linking arrow-flight-perf-server
> ---
>
> Key: ARROW-8587
> URL: https://issues.apache.org/jira/browse/ARROW-8587
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.17.0
> Environment: Linux HP 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar
> 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Chengxin Ma
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I wanted to play around with Flight benchmark after seeing the discussion
> regarding Flight's throughput in arrow dev mailing list today.
> I met the following error when trying to build the benchmark from latest
> source code:
> {code:java}
> [ 95%] Linking CXX executable ../../../debug/arrow-flight-perf-server
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::canonical(boost::filesystem::path const&,
> boost::filesystem::path const&, boost::system::error_code*)'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::system_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::parent_path() const'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::generic_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::current_path(boost::system::error_code*)'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `inflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `deflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::operator/=(boost::filesystem::path const&)'
> collect2: error: ld returned 1 exit status
> src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:154:
> recipe for target 'debug/arrow-flight-perf-server' failed
> make[2]: *** [debug/arrow-flight-perf-server] Error 1
> CMakeFiles/Makefile2:2609: recipe for target
> 'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
> make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
> Error 2
> Makefile:140: recipe for target 'all' failed
> make: *** [all] Error 2
> {code}
> I was using {{cmake .. -DCMAKE_BUILD_TYPE=Debug
> -DARROW_DEPENDENCY_SOURCE=AUTO -DARROW_FLIGHT=ON -DARROW_BUILD_BENCHMARKS=ON
> -DARROW_CXXFLAGS="-lboost_filesystem -lboost_system"}} to configure the build.
> I noticed that there was a {{ARROW_BOOST_BUILD_VERSION: 1.71.0}} in the
> output, but the Boost library that I installed from the package manger was of
> this version: {{1.65.1.0ubuntu1}}. Could this be the cause of the problem?
> PS:
> I was able to build the benchmark
> [before|https://issues.apache.org/jira/browse/ARROW-7200]. It was on AWS with
> the OS being ubuntu-bionic-18.04-amd64-server-20191002, which should be very
> similar to the one I'm using on my laptop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (ARROW-8587) Compilation error when linking arrow-flight-perf-server
[
https://issues.apache.org/jira/browse/ARROW-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxin Ma reopened ARROW-8587:
> Compilation error when linking arrow-flight-perf-server
> ---
>
> Key: ARROW-8587
> URL: https://issues.apache.org/jira/browse/ARROW-8587
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.17.0
> Environment: Linux HP 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar
> 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Chengxin Ma
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I wanted to play around with Flight benchmark after seeing the discussion
> regarding Flight's throughput in arrow dev mailing list today.
> I met the following error when trying to build the benchmark from latest
> source code:
> {code:java}
> [ 95%] Linking CXX executable ../../../debug/arrow-flight-perf-server
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::canonical(boost::filesystem::path const&,
> boost::filesystem::path const&, boost::system::error_code*)'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::system_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::parent_path() const'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::generic_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::current_path(boost::system::error_code*)'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `inflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `deflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::operator/=(boost::filesystem::path const&)'
> collect2: error: ld returned 1 exit status
> src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:154:
> recipe for target 'debug/arrow-flight-perf-server' failed
> make[2]: *** [debug/arrow-flight-perf-server] Error 1
> CMakeFiles/Makefile2:2609: recipe for target
> 'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
> make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
> Error 2
> Makefile:140: recipe for target 'all' failed
> make: *** [all] Error 2
> {code}
> I was using {{cmake .. -DCMAKE_BUILD_TYPE=Debug
> -DARROW_DEPENDENCY_SOURCE=AUTO -DARROW_FLIGHT=ON -DARROW_BUILD_BENCHMARKS=ON
> -DARROW_CXXFLAGS="-lboost_filesystem -lboost_system"}} to configure the build.
> I noticed that there was a {{ARROW_BOOST_BUILD_VERSION: 1.71.0}} in the
> output, but the Boost library that I installed from the package manger was of
> this version: {{1.65.1.0ubuntu1}}. Could this be the cause of the problem?
> PS:
> I was able to build the benchmark
> [before|https://issues.apache.org/jira/browse/ARROW-7200]. It was on AWS with
> the OS being ubuntu-bionic-18.04-amd64-server-20191002, which should be very
> similar to the one I'm using on my laptop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-8587) Compilation error when linking arrow-flight-perf-server
[
https://issues.apache.org/jira/browse/ARROW-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091863#comment-17091863
]
Chengxin Ma commented on ARROW-8587:
Thanks for the quick fix.
Unfortunately I still see the following error messages:
{code}
[ 96%] Linking CXX executable ../../../release/arrow-flight-perf-server
../../../release/libarrow_flight.so.18.0.0: undefined reference to
`inflateInit2_'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `inflate'
../../../release/libarrow_flight.so.18.0.0: undefined reference to
`deflateInit2_'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `deflate'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
../../../release/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
collect2: error: ld returned 1 exit status
src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:156: recipe
for target 'release/arrow-flight-perf-server' failed
make[2]: *** [release/arrow-flight-perf-server] Error 1
CMakeFiles/Makefile2:2648: recipe for target
'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
{code}
This seems to be a problem related to {{zlib}}. On my computer it is the latest
version: {{zlib1g-dev is already the newest version (1:1.2.11.dfsg-0ubuntu2).}}
I guess this issue is still related to {{ThirdpartyToolchain.cmake}}?
> Compilation error when linking arrow-flight-perf-server
> ---
>
> Key: ARROW-8587
> URL: https://issues.apache.org/jira/browse/ARROW-8587
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.17.0
> Environment: Linux HP 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar
> 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Chengxin Ma
>Assignee: Antoine Pitrou
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I wanted to play around with Flight benchmark after seeing the discussion
> regarding Flight's throughput in arrow dev mailing list today.
> I met the following error when trying to build the benchmark from latest
> source code:
> {code:java}
> [ 95%] Linking CXX executable ../../../debug/arrow-flight-perf-server
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::canonical(boost::filesystem::path const&,
> boost::filesystem::path const&, boost::system::error_code*)'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::system_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::parent_path() const'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::generic_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::current_path(boost::system::error_code*)'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `inflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `deflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::operator/=(boost::filesystem::path const&)'
> collect2: error: ld returned 1 exit status
> src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:154:
> recipe for target 'debug/arrow-flight-perf-server' failed
> make[2]: *** [debug/arrow-flight-perf-server] Error 1
> CMakeFiles/Makefile2:2609: recipe for target
> 'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
> make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
> Error 2
> Makefile:140: recipe for target 'all' failed
> make: *** [all] Error 2
> {code}
> I was using {{cmake .. -DCMAKE_BUILD_TYPE=Debug
> -DARROW_DEPENDENCY_SOURCE=AUTO -DARROW_FLIGHT=ON -DARROW_BUILD_BENCHMARKS=ON
> -DARROW_CXXFLAGS="-lboost_filesystem -lboost_system"}} to configure the build.
> I noticed that there was a {{ARROW_BOOST_BUILD_VERSION: 1.71.0}} in the
> output, but the Boost library that I installed from the package manger was of
> this version: {{1.65.1.0ubuntu1}}. Could this be the cause of the problem?
[jira] [Commented] (ARROW-8587) Compilation error when linking arrow-flight-perf-server
[
https://issues.apache.org/jira/browse/ARROW-8587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091761#comment-17091761
]
Chengxin Ma commented on ARROW-8587:
Additional information: I still saw this error after rolling back the code base
by: {{git checkout apache-arrow-0.17.0}}
> Compilation error when linking arrow-flight-perf-server
> ---
>
> Key: ARROW-8587
> URL: https://issues.apache.org/jira/browse/ARROW-8587
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.17.0
> Environment: Linux HP 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar
> 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Chengxin Ma
>Priority: Minor
>
> I wanted to play around with Flight benchmark after seeing the discussion
> regarding Flight's throughput in arrow dev mailing list today.
> I met the following error when trying to build the benchmark from latest
> source code:
> {code:java}
> [ 95%] Linking CXX executable ../../../debug/arrow-flight-perf-server
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::canonical(boost::filesystem::path const&,
> boost::filesystem::path const&, boost::system::error_code*)'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::system_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::parent_path() const'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::system::generic_category()'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::detail::current_path(boost::system::error_code*)'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `inflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflate'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to
> `deflateInit2_'
> ../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
> ../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
> `boost::filesystem::path::operator/=(boost::filesystem::path const&)'
> collect2: error: ld returned 1 exit status
> src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:154:
> recipe for target 'debug/arrow-flight-perf-server' failed
> make[2]: *** [debug/arrow-flight-perf-server] Error 1
> CMakeFiles/Makefile2:2609: recipe for target
> 'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
> make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
> Error 2
> Makefile:140: recipe for target 'all' failed
> make: *** [all] Error 2
> {code}
> I was using {{cmake .. -DCMAKE_BUILD_TYPE=Debug
> -DARROW_DEPENDENCY_SOURCE=AUTO -DARROW_FLIGHT=ON -DARROW_BUILD_BENCHMARKS=ON
> -DARROW_CXXFLAGS="-lboost_filesystem -lboost_system"}} to configure the build.
> I noticed that there was a {{ARROW_BOOST_BUILD_VERSION: 1.71.0}} in the
> output, but the Boost library that I installed from the package manger was of
> this version: {{1.65.1.0ubuntu1}}. Could this be the cause of the problem?
> PS:
> I was able to build the benchmark
> [before|https://issues.apache.org/jira/browse/ARROW-7200]. It was on AWS with
> the OS being ubuntu-bionic-18.04-amd64-server-20191002, which should be very
> similar to the one I'm using on my laptop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-8587) Compilation error when linking arrow-flight-perf-server
Chengxin Ma created ARROW-8587:
--
Summary: Compilation error when linking arrow-flight-perf-server
Key: ARROW-8587
URL: https://issues.apache.org/jira/browse/ARROW-8587
Project: Apache Arrow
Issue Type: Bug
Components: Benchmarking, C++, FlightRPC
Affects Versions: 1.0.0
Environment: Linux HP 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar
31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Reporter: Chengxin Ma
I wanted to play around with Flight benchmark after seeing the discussion
regarding Flight's throughput in arrow dev mailing list today.
I met the following error when trying to build the benchmark from latest source
code:
{code:java}
[ 95%] Linking CXX executable ../../../debug/arrow-flight-perf-server
../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
`boost::filesystem::detail::canonical(boost::filesystem::path const&,
boost::filesystem::path const&, boost::system::error_code*)'
../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
`boost::system::system_category()'
../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
`boost::filesystem::path::parent_path() const'
../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflate'
../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateEnd'
../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
`boost::system::generic_category()'
../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
`boost::filesystem::detail::current_path(boost::system::error_code*)'
../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateInit2_'
../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflate'
../../../debug/libarrow_flight.so.18.0.0: undefined reference to `deflateInit2_'
../../../debug/libarrow_flight.so.18.0.0: undefined reference to `inflateEnd'
../../../debug/libarrow_flight_testing.so.18.0.0: undefined reference to
`boost::filesystem::path::operator/=(boost::filesystem::path const&)'
collect2: error: ld returned 1 exit status
src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/build.make:154: recipe
for target 'debug/arrow-flight-perf-server' failed
make[2]: *** [debug/arrow-flight-perf-server] Error 1
CMakeFiles/Makefile2:2609: recipe for target
'src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all' failed
make[1]: *** [src/arrow/flight/CMakeFiles/arrow-flight-perf-server.dir/all]
Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
{code}
I was using {{cmake .. -DCMAKE_BUILD_TYPE=Debug -DARROW_DEPENDENCY_SOURCE=AUTO
-DARROW_FLIGHT=ON -DARROW_BUILD_BENCHMARKS=ON
-DARROW_CXXFLAGS="-lboost_filesystem -lboost_system"}} to configure the build.
I noticed that there was a {{ARROW_BOOST_BUILD_VERSION: 1.71.0}} in the
output, but the Boost library that I installed from the package manger was of
this version: {{1.65.1.0ubuntu1}}. Could this be the cause of the problem?
PS:
I was able to build the benchmark
[before|https://issues.apache.org/jira/browse/ARROW-7200]. It was on AWS with
the OS being ubuntu-bionic-18.04-amd64-server-20191002, which should be very
similar to the one I'm using on my laptop.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7522) Broken Record Batch returned from a function call
Chengxin Ma created ARROW-7522:
--
Summary: Broken Record Batch returned from a function call
Key: ARROW-7522
URL: https://issues.apache.org/jira/browse/ARROW-7522
Project: Apache Arrow
Issue Type: Bug
Components: C++, C++ - Plasma
Affects Versions: 0.15.1
Environment: macOS
Reporter: Chengxin Ma
Scenario: retrieving Record Batch from Plasma with known Object ID.
The following code snippet works well:
{code:java}
int main(int argc, char **argv)
{
plasma::ObjectID object_id =
plasma::ObjectID::from_binary("0FF1CE00C0FFEE00BEEF");
// Start up and connect a Plasma client.
plasma::PlasmaClient client;
ARROW_CHECK_OK(client.Connect("/tmp/store"));
plasma::ObjectBuffer object_buffer;
ARROW_CHECK_OK(client.Get(_id, 1, -1, _buffer));
// Retrieve object data.
auto buffer = object_buffer.data;
arrow::io::BufferReader buffer_reader(buffer);
std::shared_ptr record_batch_stream_reader;
ARROW_CHECK_OK(arrow::ipc::RecordBatchStreamReader::Open(_reader,
_batch_stream_reader));
std::shared_ptr record_batch;
arrow::Status status = record_batch_stream_reader->ReadNext(_batch);
std::cout << "record_batch->column_name(0): " <<
record_batch->column_name(0) << std::endl;
std::cout << "record_batch->num_columns(): " << record_batch->num_columns()
<< std::endl;
std::cout << "record_batch->num_rows(): " << record_batch->num_rows() <<
std::endl;
std::cout << "record_batch->column(0)->length(): "
<< record_batch->column(0)->length() << std::endl;
std::cout << "record_batch->column(0)->ToString(): "
<< record_batch->column(0)->ToString() << std::endl;
}
{code}
{{record_batch->column(0)->ToString()}} would incur a segmentation fault if
retrieving Record Batch is wrapped in a function:
{code:java}
std::shared_ptr GetRecordBatchFromPlasma(plasma::ObjectID
object_id)
{
// Start up and connect a Plasma client.
plasma::PlasmaClient client;
ARROW_CHECK_OK(client.Connect("/tmp/store"));
plasma::ObjectBuffer object_buffer;
ARROW_CHECK_OK(client.Get(_id, 1, -1, _buffer));
// Retrieve object data.
auto buffer = object_buffer.data;
arrow::io::BufferReader buffer_reader(buffer);
std::shared_ptr record_batch_stream_reader;
ARROW_CHECK_OK(arrow::ipc::RecordBatchStreamReader::Open(_reader,
_batch_stream_reader));
std::shared_ptr record_batch;
arrow::Status status = record_batch_stream_reader->ReadNext(_batch);
// Disconnect the client.
ARROW_CHECK_OK(client.Disconnect());
return record_batch;
}
int main(int argc, char **argv)
{
plasma::ObjectID object_id =
plasma::ObjectID::from_binary("0FF1CE00C0FFEE00BEEF");
std::shared_ptr record_batch =
GetRecordBatchFromPlasma(object_id);
std::cout << "record_batch->column_name(0): " <<
record_batch->column_name(0) << std::endl;
std::cout << "record_batch->num_columns(): " << record_batch->num_columns()
<< std::endl;
std::cout << "record_batch->num_rows(): " << record_batch->num_rows() <<
std::endl;
std::cout << "record_batch->column(0)->length(): "
<< record_batch->column(0)->length() << std::endl;
std::cout << "record_batch->column(0)->ToString(): "
<< record_batch->column(0)->ToString() << std::endl;
}
{code}
The meta info of the Record Batch such as number of columns and rows is still
available, but I can't see the content of the columns.
{{lldb}} says that the stop reason is {{EXC_BAD_ACCESS}}, so I think the Record
Batch is destroyed after {{GetRecordBatchFromPlasma}} finishes. But why can I
still see the meta info of this Record Batch?
What is the proper way to get the Record Batch if we insist using a function?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (ARROW-7434) [GLib] Homebrew packages seem not working
[
https://issues.apache.org/jira/browse/ARROW-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxin Ma closed ARROW-7434.
--
Resolution: Not A Bug
> [GLib] Homebrew packages seem not working
> -
>
> Key: ARROW-7434
> URL: https://issues.apache.org/jira/browse/ARROW-7434
> Project: Apache Arrow
> Issue Type: Bug
> Components: GLib
>Affects Versions: 0.15.1
> Environment: macOS 10.15.2
>Reporter: Chengxin Ma
>Priority: Major
>
> After installing {{apache-arrow}} and {{apache-arrow-glib}} via {{Homebrew}}
> according to the [Installation Guide|https://arrow.apache.org/install/], I
> wrote a very simple program to test if they were successfully installed.
> {code}
> $ cat hello_world.c
> #include
> #include
> int main(int argc, char **argv) {
> printf("Hello, World! \n");
> }
> {code}
> {{gcc}} gave the following error:
> {code}
> $ gcc -o hello_world hello_world.c
> In file included from hello_world.c:3:
> In file included from /usr/local/include/arrow-glib/arrow-glib.h:22:
> /usr/local/include/arrow-glib/gobject-type.h:22:10: fatal error:
> 'glib-object.h' file not found
> #include
> ^~~
> 1 error generated.
> {code}
> Is there any step that I didn’t follow here?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7434) [GLib] Homebrew packages seem not working
[
https://issues.apache.org/jira/browse/ARROW-7434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16999207#comment-16999207
]
Chengxin Ma commented on ARROW-7434:
Solution from the dev mailing list: {{gcc -o hello_world hello_world.c
$(pkg-config --libs --cflags arrow-glib)}}
> [GLib] Homebrew packages seem not working
> -
>
> Key: ARROW-7434
> URL: https://issues.apache.org/jira/browse/ARROW-7434
> Project: Apache Arrow
> Issue Type: Bug
> Components: GLib
>Affects Versions: 0.15.1
> Environment: macOS 10.15.2
>Reporter: Chengxin Ma
>Priority: Major
>
> After installing {{apache-arrow}} and {{apache-arrow-glib}} via {{Homebrew}}
> according to the [Installation Guide|https://arrow.apache.org/install/], I
> wrote a very simple program to test if they were successfully installed.
> {code}
> $ cat hello_world.c
> #include
> #include
> int main(int argc, char **argv) {
> printf("Hello, World! \n");
> }
> {code}
> {{gcc}} gave the following error:
> {code}
> $ gcc -o hello_world hello_world.c
> In file included from hello_world.c:3:
> In file included from /usr/local/include/arrow-glib/arrow-glib.h:22:
> /usr/local/include/arrow-glib/gobject-type.h:22:10: fatal error:
> 'glib-object.h' file not found
> #include
> ^~~
> 1 error generated.
> {code}
> Is there any step that I didn’t follow here?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7434) [GLib] Homebrew packages seem not working
Chengxin Ma created ARROW-7434:
--
Summary: [GLib] Homebrew packages seem not working
Key: ARROW-7434
URL: https://issues.apache.org/jira/browse/ARROW-7434
Project: Apache Arrow
Issue Type: Bug
Components: GLib
Affects Versions: 0.15.1
Environment: macOS 10.15.2
Reporter: Chengxin Ma
After installing {{apache-arrow}} and {{apache-arrow-glib}} via {{Homebrew}}
according to the [Installation Guide|https://arrow.apache.org/install/], I
wrote a very simple program to test if they were successfully installed.
{code}
$ cat hello_world.c
#include
#include
int main(int argc, char **argv) {
printf("Hello, World! \n");
}
{code}
{{gcc}} gave the following error:
{code}
$ gcc -o hello_world hello_world.c
In file included from hello_world.c:3:
In file included from /usr/local/include/arrow-glib/arrow-glib.h:22:
/usr/local/include/arrow-glib/gobject-type.h:22:10: fatal error:
'glib-object.h' file not found
#include
^~~
1 error generated.
{code}
Is there any step that I didn’t follow here?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7411) [C++][Flight] Incorrect Arrow Flight benchmark output
Chengxin Ma created ARROW-7411:
--
Summary: [C++][Flight] Incorrect Arrow Flight benchmark output
Key: ARROW-7411
URL: https://issues.apache.org/jira/browse/ARROW-7411
Project: Apache Arrow
Issue Type: Improvement
Components: Benchmarking, C++, FlightRPC
Affects Versions: 0.15.1
Environment: macOS
Reporter: Chengxin Ma
Assignee: Chengxin Ma
Fix For: 1.0.0
When running Arrow Flight benchmark in the following scenario, the output is
incorrect.
{code}
$ ./arrow-flight-perf-server &
[1] 12986
Server host: localhost
Server port: 31337
$ ./arrow-flight-benchmark -server_host localhost -test_put
Using remote server: true
Testing method: DoPut
Server host: localhost
Server port: 31337
Bytes read: 128000
Nanos: 496372147
Speed: 2459.25 MB/s
{code}
{{Using remote server}} should be {{false}} and {{Bytes read}} should be
{{Bytes write}}.
To correct the result of {{Using remote server}}, we can:
* Change {{if (FLAGS_server_host == "")}} to another condition which checks if
there is already an {{arrow-flight-perf-server}} running. This is a bit
complicated to do and might add some unnecessary complexity (e.g. we need to
make sure it support all OSes.).
* Delete {{Using remote server}}, since we already have {{Server host}} in the
output.
I personally prefer the second option and will make a PR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ARROW-7320) [C++] Target arrow-type-benchmark failed to be built on bullx Linux
[
https://issues.apache.org/jira/browse/ARROW-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxin Ma reassigned ARROW-7320:
--
Assignee: Chengxin Ma
> [C++] Target arrow-type-benchmark failed to be built on bullx Linux
> ---
>
> Key: ARROW-7320
> URL: https://issues.apache.org/jira/browse/ARROW-7320
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 1.0.0
> Environment: bullx Linux
>Reporter: Chengxin Ma
>Assignee: Chengxin Ma
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I was building Arrow on bullx Linux (a Linux distribution compatible with Red
> Hat Enterprise Linux).
> CMake options:
> {code}
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_FLIGHT=ON
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
> {{make}} failed with the following error message:
> {code}
> Scanning dependencies of target arrow-type-benchmark
> [ 72%] Building CXX object
> src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
> make[2]: *** No rule to make target
> `gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
> `debug/arrow-type-benchmark'. Stop.
> make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
> make: *** [all] Error 2
> {code}
> This is due to the same reason as mentioned in [this
> commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
> If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
> {{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7320) Target arrow-type-benchmark failed to be built on bullx Linux
Chengxin Ma created ARROW-7320:
--
Summary: Target arrow-type-benchmark failed to be built on bullx
Linux
Key: ARROW-7320
URL: https://issues.apache.org/jira/browse/ARROW-7320
Project: Apache Arrow
Issue Type: Bug
Components: C++
Affects Versions: 1.0.0
Environment: bullx Linux
Reporter: Chengxin Ma
I was building Arrow on bullx Linux (a Linux distribution compatible with Red
Hat Enterprise Linux).
CMake options:
{code}
-DCMAKE_BUILD_TYPE=Debug
-DARROW_FLIGHT=ON
-DARROW_BUILD_BENCHMARKS=ON
{code}
{{make}} failed with the following error message:
{code}
Scanning dependencies of target arrow-type-benchmark
[ 72%] Building CXX object
src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
make[2]: *** No rule to make target
`gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
`debug/arrow-type-benchmark'. Stop.
make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
make: *** [all] Error 2
{code}
This is due to the same reason as mentioned in [this
commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
{{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7320) [C++] Target arrow-type-benchmark failed to be built on bullx Linux
[
https://issues.apache.org/jira/browse/ARROW-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxin Ma updated ARROW-7320:
---
Summary: [C++] Target arrow-type-benchmark failed to be built on bullx
Linux (was: Target arrow-type-benchmark failed to be built on bullx Linux)
> [C++] Target arrow-type-benchmark failed to be built on bullx Linux
> ---
>
> Key: ARROW-7320
> URL: https://issues.apache.org/jira/browse/ARROW-7320
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Affects Versions: 1.0.0
> Environment: bullx Linux
>Reporter: Chengxin Ma
>Priority: Major
>
> I was building Arrow on bullx Linux (a Linux distribution compatible with Red
> Hat Enterprise Linux).
> CMake options:
> {code}
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_FLIGHT=ON
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
> {{make}} failed with the following error message:
> {code}
> Scanning dependencies of target arrow-type-benchmark
> [ 72%] Building CXX object
> src/arrow/CMakeFiles/arrow-type-benchmark.dir/type_benchmark.cc.o
> make[2]: *** No rule to make target
> `gbenchmark_ep/src/gbenchmark_ep-install/lib/libbenchmark_main.a', needed by
> `debug/arrow-type-benchmark'. Stop.
> make[1]: *** [src/arrow/CMakeFiles/arrow-type-benchmark.dir/all] Error 2
> make: *** [all] Error 2
> {code}
> This is due to the same reason as mentioned in [this
> commit|https://github.com/apache/arrow/pull/4246/commits/f6b0bc7f8dc56f02e2778752235e728b7623a9ee]:
> If {{-DCMAKE_INSTALL_LIBDIR=lib}} is not explicitly set,
> {{libbenchmark_main.a}} will be put in {{lib64}} instead of {{lib}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7200) [C++][Flight] Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980743#comment-16980743
]
Chengxin Ma commented on ARROW-7200:
I've attempted to do a fix here:
[https://github.com/MaChengxin/arrow/commit/08ff56bacd87accb52adc2a20f1976acb320917e]
I'm not sure what the correct workflow is. Is it PR-then-code-review or the
other way around?
Now in order to enable benchmarking on two hosts, we have to do:
Server side:
{code}
ubuntu@ip-172-31-11-18:~/arrow/cpp/for_flight/debug$ ./arrow-flight-perf-server
-server_host 172.31.11.18
Server host: 172.31.11.18
Server port: 31337
{code}
Client side:
{code}
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ ./arrow-flight-benchmark
-server_host 172.31.11.18
Using remote server: true
Testing method: DoGet
Server host: 172.31.11.18
Server port: 31337
Bytes read: 128000
Nanos: 2080762694
Speed: 586.661 MB/s
{code}
If we don't specify the {{server_host}} flag on the server side, it will
explicitly says it's serving locally:
{code}
ubuntu@ip-172-31-11-18:~/arrow/cpp/for_flight/debug$ ./arrow-flight-perf-server
Server host: localhost
Server port: 31337
{code}
> [C++][Flight] Running Arrow Flight benchmark on two hosts doesn't work
> --
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the problem is in the environment or in the code itself.
> Could someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979268#comment-16979268
]
Chengxin Ma commented on ARROW-7200:
I did a quick fix on this issue:
[https://github.com/MaChengxin/arrow/commit/01befee9ca8da0ae8f514e206665a3b2a411aa71?diff=split]
I'm not sure if this will introduce other problems, but it looks like it is
working properly:
{code}
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ ./arrow-flight-benchmark
-server_host 172.31.11.18
Using remote server: true
Testing method: DoGet
Server host: 172.31.11.18
Server port: 31337
Bytes read: 128000
Nanos: 2113874359
Speed: 577.472 MB/s
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ ./arrow-flight-benchmark
-server_host 172.31.11.18 -test_put true
Using remote server: true
Testing method: DoPut
Server host: 172.31.11.18
Server port: 31337
Bytes read: 128000
Nanos: 2164233449
Speed: 564.035 MB/s
{code}
> Running Arrow Flight benchmark on two hosts doesn't work
>
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the problem is in the environment or in the code itself.
> Could someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979172#comment-16979172
]
Chengxin Ma commented on ARROW-7200:
gRPC verbose logging shows:
{code:java}
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ env GRPC_VERBOSITY=debug
GRPC_TRACE=subchannel ./arrow-flight-benchmark -server_host 172.31.11.18
Using remote server: true
Testing method: DoGet
Server host: 172.31.11.18
Server port: 31337
...
I1121 09:28:58.051069998 14713 subchannel.cc:960] Connect failed:
{"created":"@1574328538.050990277","description":"Failed to connect to remote
host: Connection
refused","errno":111,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":207,"os_error":"Connection
refused","syscall":"connect","target_address":"ipv6:[::1]:31337"}
I1121 09:28:58.051209019 14713 subchannel.cc:960] Connect failed:
{"created":"@1574328538.051185929","description":"Failed to connect to remote
host: Connection
refused","errno":111,"file":"src/core/lib/iomgr/tcp_client_posix.cc","file_line":207,"os_error":"Connection
refused","syscall":"connect","target_address":"ipv4:127.0.0.1:31337"}
...
{code}
It seems that the client wanted to connect to the loopback address of the
server, which would certainly fail.
The problem is perhaps in the {{RunPerformanceTest}} function of
[flight_benchmark.cc|https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/flight/flight_benchmark.cc].
At [line
209|https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/flight/flight_benchmark.cc#L209]
the client will try to connect to the server again, but it will use
{{localhost}} instead of the server's IP address:
If I add
{code:java}
for (const auto& location : endpoint.locations) {
std::cout << location.ToString() << std::endl;
}
{code}
after [line
233|https://github.com/apache/arrow/blob/maint-0.15.x/cpp/src/arrow/flight/flight_benchmark.cc#L233],
and run the client again, it will show:
{code}
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ ./arrow-flight-benchmark
--server_host 172.31.11.18
Using remote server: true
Testing method: DoGet
Server host: 172.31.11.18
Server port: 31337
grpc+tcp://localhost:31337
grpc+tcp://localhost:31337
grpc+tcp://localhost:31337
grpc+tcp://localhost:31337
Failed with error: << IOError: gRPC returned unavailable error, with message:
Connect Failed. Detail: Unavailable
{code}
> Running Arrow Flight benchmark on two hosts doesn't work
>
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the
[jira] [Commented] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978604#comment-16978604
]
Chengxin Ma commented on ARROW-7200:
[~apitrou], Thank you, I didn't know {{netstat}} and {{nc}} before. Thanks for
letting me know.
[~lidavidm], I don't mind sharing the {{pem}} file for accessing the ad hoc AWS
instances, if you think it is more handy for you to debug (if necessary).
> Running Arrow Flight benchmark on two hosts doesn't work
>
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the problem is in the environment or in the code itself.
> Could someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978589#comment-16978589
]
Chengxin Ma commented on ARROW-7200:
I just tried:
{code}
ubuntu@ip-172-31-3-53:~$ nc -v 172.31.11.18 31337
Connection to 172.31.11.18 31337 port [tcp/*] succeeded!
@@ ^C
ubuntu@ip-172-31-3-53:~$ nc -v 172.31.11.18 22
Connection to 172.31.11.18 22 port [tcp/ssh] succeeded!
SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
{code}
I have no idea about the returned {{@@}}.
> Running Arrow Flight benchmark on two hosts doesn't work
>
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the problem is in the environment or in the code itself.
> Could someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16978537#comment-16978537
]
Chengxin Ma commented on ARROW-7200:
Yes, the two machines (hostnames: {{ip-172-31-11-18}} and {{ip-172-31-3-53}})
are in the same private network on AWS.
The output of {{/etc/hosts}} on {{ip-172-31-3-53}} is:
{code}
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ cat /etc/hosts
127.0.0.1 localhost
172.31.11.18 ip-172-31-11-18
{code}
I can ping {{ip-172-31-11-18}} from {{ip-172-31-3-53}}:
{code}
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ ping ip-172-31-11-18
PING ip-172-31-11-18 (172.31.11.18) 56(84) bytes of data.
64 bytes from ip-172-31-11-18 (172.31.11.18): icmp_seq=1 ttl=64 time=0.163 ms
64 bytes from ip-172-31-11-18 (172.31.11.18): icmp_seq=2 ttl=64 time=0.178 ms
64 bytes from ip-172-31-11-18 (172.31.11.18): icmp_seq=3 ttl=64 time=0.167 ms
64 bytes from ip-172-31-11-18 (172.31.11.18): icmp_seq=4 ttl=64 time=0.175 ms
64 bytes from ip-172-31-11-18 (172.31.11.18): icmp_seq=5 ttl=64 time=0.168 ms
^C
--- ip-172-31-11-18 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4087ms
rtt min/avg/max/mdev = 0.163/0.170/0.178/0.009 ms
{code}
I also tried to use the IP address directly as the command line argument
instead of the hostname, but it still fails.
{code}
ubuntu@ip-172-31-3-53:~/arrow/cpp/for_flight/debug$ ./arrow-flight-benchmark
-server_host 172.31.11.18
Using remote server: true
Testing method: DoGet
Server host: 172.31.11.18
Server port: 31337
Failed with error: << IOError: gRPC returned unavailable error, with message:
Connect Failed. Detail: Unavailable
{code}
> Running Arrow Flight benchmark on two hosts doesn't work
>
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the problem is in the environment or in the code itself.
> Could someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977508#comment-16977508
]
Chengxin Ma commented on ARROW-7200:
{{netstat}} shows:
{{tcp6 0 0 :::31337:::*LISTEN
3831/./arrow-flight}}
(where {{arrow-flight}} is truncated from {{arrow-flight-perf-server}}).
This looks OK.
I checked the source code of the Flight benchmark, and it seems that the error
is returned from the {{DoGet}} method.
(https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc#L92)
If I set {{--test_put}} to be {{true}}, I would get the following error:
{{Failed with error: << IOError: Could not write record batch to stream:}}
The screenshot below shows the result of testing {{DoPut}} with and without the
server running.
!Screen Shot 2019-11-19 at 14.41.40.png!
Obviously the client is aware of the existence of the server, but somehow it
can't read from nor write to the server.
What could possible cause this problem?
PS: I also tried the Hello World example of gRPC
(https://grpc.io/docs/quickstart/cpp/) with some modification (changing
localhost to the remote one), and it worked well.
> Running Arrow Flight benchmark on two hosts doesn't work
>
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the problem is in the environment or in the code itself.
> Could someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
[
https://issues.apache.org/jira/browse/ARROW-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxin Ma updated ARROW-7200:
---
Attachment: Screen Shot 2019-11-19 at 14.41.40.png
> Running Arrow Flight benchmark on two hosts doesn't work
>
>
> Key: ARROW-7200
> URL: https://issues.apache.org/jira/browse/ARROW-7200
> Project: Apache Arrow
> Issue Type: Bug
> Components: Benchmarking, C++, FlightRPC
>Affects Versions: 0.15.0, 0.15.1
> Environment: AWS EC2
> Instance type: t3a.xlarge
> AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
> Number of instances: 2
> They are capable of pinging each other.
>Reporter: Chengxin Ma
>Priority: Major
> Attachments: Screen Shot 2019-11-18 at 16.00.38.png, Screen Shot
> 2019-11-19 at 14.41.40.png
>
>
> I was trying to evaluate the performance of Apache Arrow Flight on two hosts
> (one as the client and the other one as the server), using [the official
> benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
> Flags I used to build the project were:
>
> {code:java}
> -DARROW_FLIGHT=ON
> -DCMAKE_BUILD_TYPE=Debug
> -DARROW_BUILD_BENCHMARKS=ON
> {code}
>
> The branch I used was maint-0.15.x since there was a build error on the
> master branch. _(The build error on master only existed in the environment
> where I set up two hosts: AWS. On my local environment (macOS) the build was
> successful on the master branch. I don't think this build error is relevant
> to the issue since there is no difference in the cpp source code.)_
> On the host acting as the server, I ran
> {code:java}
> ./arrow-flight-perf-server{code}
> On the host acting as the client, I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
> It gives the following error:
> {code:java}
> Failed with error: << IOError: gRPC returned unavailable error, with message:
> Connect Failed. Detail: Unavailable{code}
>
> If I ran
> {code:java}
> ./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
> the error will be different:
> {code:java}
> IOError: Server was not available after 10 attempts{code}
> This is understandable since this host doesn't exist at all.
> This indicates that Flight is able to find the existing host
> (ip-172-31-11-18), but the communication somehow didn't succeed.
> The benchmark works fine if I run it with the localhost, either by not
> specifying the server_host flag or running the server in another process on
> the same host.
> I am not sure if the problem is in the environment or in the code itself.
> Could someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (ARROW-7200) Running Arrow Flight benchmark on two hosts doesn't work
Chengxin Ma created ARROW-7200:
--
Summary: Running Arrow Flight benchmark on two hosts doesn't work
Key: ARROW-7200
URL: https://issues.apache.org/jira/browse/ARROW-7200
Project: Apache Arrow
Issue Type: Bug
Components: Benchmarking, C++, FlightRPC
Affects Versions: 0.15.1, 0.15.0
Environment: AWS EC2
Instance type: t3a.xlarge
AMI: ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002
Number of instances: 2
They are capable of pinging each other.
Reporter: Chengxin Ma
Attachments: Screen Shot 2019-11-18 at 16.00.38.png
I was trying to evaluate the performance of Apache Arrow Flight on two hosts
(one as the client and the other one as the server), using [the official
benchmark|[https://github.com/apache/arrow/blob/master/cpp/src/arrow/flight/flight_benchmark.cc]].
Flags I used to build the project were:
{code:java}
-DARROW_FLIGHT=ON
-DCMAKE_BUILD_TYPE=Debug
-DARROW_BUILD_BENCHMARKS=ON
{code}
The branch I used was maint-0.15.x since there was a build error on the master
branch. _(The build error on master only existed in the environment where I set
up two hosts: AWS. On my local environment (macOS) the build was successful on
the master branch. I don't think this build error is relevant to the issue
since there is no difference in the cpp source code.)_
On the host acting as the server, I ran
{code:java}
./arrow-flight-perf-server{code}
On the host acting as the client, I ran
{code:java}
./arrow-flight-benchmark --server_host ip-172-31-11-18{code}
It gives the following error:
{code:java}
Failed with error: << IOError: gRPC returned unavailable error, with message:
Connect Failed. Detail: Unavailable{code}
If I ran
{code:java}
./arrow-flight-benchmark --server_host ip-172-31-11-17{code}
the error will be different:
{code:java}
IOError: Server was not available after 10 attempts{code}
This is understandable since this host doesn't exist at all.
This indicates that Flight is able to find the existing host (ip-172-31-11-18),
but the communication somehow didn't succeed.
The benchmark works fine if I run it with the localhost, either by not
specifying the server_host flag or running the server in another process on the
same host.
I am not sure if the problem is in the environment or in the code itself. Could
someone please give me some hint on how to resolve the problem?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
