marin-ma opened a new pull request, #12357:
URL: https://github.com/apache/gluten/pull/12357
Several gpu shuffle writer tests failed with below error
```
[ RUN ]
HashPartitioningShuffleWriterGroup/GpuHashPartitioningShuffleWriterTest.hashPart1Vector/0
Running test with param: shuffleWriterType = gpu_hash, partitionWriterType =
LocalPartitionWriter, compressionType = uncompressed, compressionThreshold =
-1, mergeBufferSize = 0, compressionBufferSize = 0, deserializerBufferSize = 0
unknown file: Failure
C++ exception with description "CUDA error at:
/root/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/rmm-src/cpp/src/cuda_stream_view.cpp:45:
cudaErrorIllegalAddress an illegal memory access was encountered" thrown in
the test body.
[ FAILED ]
HashPartitioningShuffleWriterGroup/GpuHashPartitioningShuffleWriterTest.hashPart1Vector/0,
where GetParam() = 40-byte object <03-00 00-00 00-00 00-00 00-00 00-00 FF-FF
FF-FF 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>
(385 ms)
```
**Use cuda compute-sanitizer to diagnose**
```
/usr/local/cuda-13.1/bin/compute-sanitizer --tool memcheck \
./velox_gpu_shuffle_writer_test --gtest_filter=*hashPart1Vector/0
```
Output:
```
========= COMPUTE-SANITIZER
Note: Google Test filter = *hashPart1Vector/0
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
I20260624 10:58:25.716317 342444 VeloxBackend.cc:152] VeloxBackend config:
[spark.gluten.sql.debug, true]
[spark.gluten.memory.reservationBlockSize, 1]
W20260624 10:58:25.716445 342444 VeloxBackend.cc:216]
spark.gluten.numTaskSlotsPerExecutor is not set. Falling back to 1.
I20260624 10:58:25.749213 342444 VeloxBackend.cc:285] Setting global Velox
memory manager with capacity: 9223372036854775807
[----------] 1 test from
HashPartitioningShuffleWriterGroup/GpuHashPartitioningShuffleWriterTest
[ RUN ]
HashPartitioningShuffleWriterGroup/GpuHashPartitioningShuffleWriterTest.hashPart1Vector/0
Running test with param: shuffleWriterType = gpu_hash, partitionWriterType =
LocalPartitionWriter, compressionType = uncompressed, compressionThreshold =
-1, mergeBufferSize = 0, compressionBufferSize = 0, deserializerBufferSize = 0
========= Invalid __global__ read of size 4 bytes
========= at void
cudf::detail::<unnamed>::count_set_bits_kernel<(int)256>(cuda::std::__4::span<const
unsigned int *const , (unsigned long)18446744073709551615>, int, int, int
*)+0x1180
========= by thread (0,0,0) in block (0,0,0)
========= Access to 0x360d680 is out of bounds
========= and is 140,268,057,602,432 bytes before the nearest allocation
at 0x7f92b7200000 of size 2 bytes
========= Saved host backtrace up to driver entry point at kernel launch
time
========= Host Frame: cudaLaunchKernel [0x3d45137] in libcudf.so
========= Host Frame: cudf::detail::(anonymous
namespace)::batch_count_set_bits(cudf::host_span<unsigned int const* const,
18446744073709551615ul>, int, int, rmm::cuda_stream_view) [0xa3af01] in
libcudf.so
========= Host Frame: cudf::detail::count_set_bits(unsigned int
const*, int, int, rmm::cuda_stream_view) [0xa3b210] in libcudf.so
========= Host Frame: cudf::null_count(unsigned int const*, int,
int, rmm::cuda_stream_view) [0xa3b57e] in libcudf.so
========= Host Frame: std::unique_ptr<cudf::column,
std::default_delete<cudf::column> > gluten::(anonymous
namespace)::DispatchColumn::readFlatColumn<(facebook::velox::TypeKind)5,
float>(cudf::type_id) [0x271e542] in libvelox.so
========= Host Frame: gluten::GpuBufferBatchResizer::next()
[0x271f086] in libvelox.so
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::shuffleWriteReadMultiBlocks(gluten::VeloxShuffleWriter&,
int, std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > > const&)
[0x12eeae] in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::testShuffleRoundTrip(gluten::VeloxShuffleWriter&,
std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >, int,
std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > >) [0x130786]
in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuHashPartitioningShuffleWriterTest_hashPart1Vector_Test::TestBody()
[0x1016ce] in velox_gpu_shuffle_writer_test
========= Host Frame: void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*) [0x29bbcc] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::Test::Run() [clone .part.0]
[0x29273b] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestInfo::Run() [0x292c51] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestSuite::Run() [clone .part.0]
[0x293160] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::internal::UnitTestImpl::RunAllTests()
[0x294320] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::UnitTest::Run() [0x2949df] in
velox_gpu_shuffle_writer_test
========= Host Frame: main [0xee986] in velox_gpu_shuffle_writer_test
=========
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified
launch failure" on CUDA API call to cudaStreamSynchronize.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: rmm::cuda_stream_view::synchronize() const
[0x29d038] in velox_gpu_shuffle_writer_test
========= Host Frame: cudf::detail::host_vector<int>
cudf::detail::make_pinned_vector_async<int>(unsigned long,
rmm::cuda_stream_view) [0xa45637] in libcudf.so
========= Host Frame: cudf::detail::(anonymous
namespace)::batch_count_set_bits(cudf::host_span<unsigned int const* const,
18446744073709551615ul>, int, int, rmm::cuda_stream_view) [0xa3af1c] in
libcudf.so
========= Host Frame: cudf::detail::count_set_bits(unsigned int
const*, int, int, rmm::cuda_stream_view) [0xa3b210] in libcudf.so
========= Host Frame: cudf::null_count(unsigned int const*, int,
int, rmm::cuda_stream_view) [0xa3b57e] in libcudf.so
========= Host Frame: std::unique_ptr<cudf::column,
std::default_delete<cudf::column> > gluten::(anonymous
namespace)::DispatchColumn::readFlatColumn<(facebook::velox::TypeKind)5,
float>(cudf::type_id) [0x271e542] in libvelox.so
========= Host Frame: gluten::GpuBufferBatchResizer::next()
[0x271f086] in libvelox.so
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::shuffleWriteReadMultiBlocks(gluten::VeloxShuffleWriter&,
int, std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > > const&)
[0x12eeae] in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::testShuffleRoundTrip(gluten::VeloxShuffleWriter&,
std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >, int,
std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > >) [0x130786]
in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuHashPartitioningShuffleWriterTest_hashPart1Vector_Test::TestBody()
[0x1016ce] in velox_gpu_shuffle_writer_test
========= Host Frame: void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*) [0x29bbcc] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::Test::Run() [clone .part.0]
[0x29273b] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestInfo::Run() [0x292c51] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestSuite::Run() [clone .part.0]
[0x293160] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::internal::UnitTestImpl::RunAllTests()
[0x294320] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::UnitTest::Run() [0x2949df] in
velox_gpu_shuffle_writer_test
========= Host Frame: main [0xee986] in velox_gpu_shuffle_writer_test
=========
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified
launch failure" on CUDA API call to cudaGetLastError.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: rmm::cuda_stream_view::synchronize() const
[0x29d055] in velox_gpu_shuffle_writer_test
========= Host Frame: cudf::detail::host_vector<int>
cudf::detail::make_pinned_vector_async<int>(unsigned long,
rmm::cuda_stream_view) [0xa45637] in libcudf.so
========= Host Frame: cudf::detail::(anonymous
namespace)::batch_count_set_bits(cudf::host_span<unsigned int const* const,
18446744073709551615ul>, int, int, rmm::cuda_stream_view) [0xa3af1c] in
libcudf.so
========= Host Frame: cudf::detail::count_set_bits(unsigned int
const*, int, int, rmm::cuda_stream_view) [0xa3b210] in libcudf.so
========= Host Frame: cudf::null_count(unsigned int const*, int,
int, rmm::cuda_stream_view) [0xa3b57e] in libcudf.so
========= Host Frame: std::unique_ptr<cudf::column,
std::default_delete<cudf::column> > gluten::(anonymous
namespace)::DispatchColumn::readFlatColumn<(facebook::velox::TypeKind)5,
float>(cudf::type_id) [0x271e542] in libvelox.so
========= Host Frame: gluten::GpuBufferBatchResizer::next()
[0x271f086] in libvelox.so
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::shuffleWriteReadMultiBlocks(gluten::VeloxShuffleWriter&,
int, std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > > const&)
[0x12eeae] in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::testShuffleRoundTrip(gluten::VeloxShuffleWriter&,
std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >, int,
std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > >) [0x130786]
in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuHashPartitioningShuffleWriterTest_hashPart1Vector_Test::TestBody()
[0x1016ce] in velox_gpu_shuffle_writer_test
========= Host Frame: void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*) [0x29bbcc] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::Test::Run() [clone .part.0]
[0x29273b] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestInfo::Run() [0x292c51] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestSuite::Run() [clone .part.0]
[0x293160] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::internal::UnitTestImpl::RunAllTests()
[0x294320] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::UnitTest::Run() [0x2949df] in
velox_gpu_shuffle_writer_test
========= Host Frame: main [0xee986] in velox_gpu_shuffle_writer_test
=========
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified
launch failure" on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: void
cuda::__4::__override_fn_<rmm::mr::cuda_memory_resource, &(decltype
(({parm#1}.deallocate)({parm#2}, {parm#3}, {parm#4}, {parm#5}))
cuda::mr::__4::__deallocate_async<rmm::mr::cuda_memory_resource>(rmm::mr::cuda_memory_resource&,
cuda::__4::stream_ref, void*, unsigned long, unsigned long)), void, false,
false, cuda::__4::stream_ref, void*, unsigned long, unsigned
long>(cuda::std::__4::conditional<false, void const, void>::type*,
cuda::__4::stream_ref, void*, unsigned long, unsigned long) [0x271f8cb] in
libvelox.so
========= Host Frame: rmm::device_buffer::deallocate_async()
[0x29e7ea] in velox_gpu_shuffle_writer_test
========= Host Frame: rmm::device_buffer::~device_buffer()
[0x29e8e8] in velox_gpu_shuffle_writer_test
========= Host Frame: cudf::detail::(anonymous
namespace)::batch_count_set_bits(cudf::host_span<unsigned int const* const,
18446744073709551615ul>, int, int, rmm::cuda_stream_view) [clone .cold]
[0x301bb8] in libcudf.so
========= Host Frame: cudf::detail::count_set_bits(unsigned int
const*, int, int, rmm::cuda_stream_view) [0xa3b210] in libcudf.so
========= Host Frame: cudf::null_count(unsigned int const*, int,
int, rmm::cuda_stream_view) [0xa3b57e] in libcudf.so
========= Host Frame: std::unique_ptr<cudf::column,
std::default_delete<cudf::column> > gluten::(anonymous
namespace)::DispatchColumn::readFlatColumn<(facebook::velox::TypeKind)5,
float>(cudf::type_id) [0x271e542] in libvelox.so
========= Host Frame: gluten::GpuBufferBatchResizer::next()
[0x271f086] in libvelox.so
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::shuffleWriteReadMultiBlocks(gluten::VeloxShuffleWriter&,
int, std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > > const&)
[0x12eeae] in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::testShuffleRoundTrip(gluten::VeloxShuffleWriter&,
std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >, int,
std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > >) [0x130786]
in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuHashPartitioningShuffleWriterTest_hashPart1Vector_Test::TestBody()
[0x1016ce] in velox_gpu_shuffle_writer_test
========= Host Frame: void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*) [0x29bbcc] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::Test::Run() [clone .part.0]
[0x29273b] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestInfo::Run() [0x292c51] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestSuite::Run() [clone .part.0]
[0x293160] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::internal::UnitTestImpl::RunAllTests()
[0x294320] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::UnitTest::Run() [0x2949df] in
velox_gpu_shuffle_writer_test
========= Host Frame: main [0xee986] in velox_gpu_shuffle_writer_test
=========
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified
launch failure" on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: void
cuda::__4::__override_fn_<rmm::mr::cuda_memory_resource, &(decltype
(({parm#1}.deallocate)({parm#2}, {parm#3}, {parm#4}, {parm#5}))
cuda::mr::__4::__deallocate_async<rmm::mr::cuda_memory_resource>(rmm::mr::cuda_memory_resource&,
cuda::__4::stream_ref, void*, unsigned long, unsigned long)), void, false,
false, cuda::__4::stream_ref, void*, unsigned long, unsigned
long>(cuda::std::__4::conditional<false, void const, void>::type*,
cuda::__4::stream_ref, void*, unsigned long, unsigned long) [0x271f8cb] in
libvelox.so
========= Host Frame: rmm::device_buffer::deallocate_async()
[0x29e7ea] in velox_gpu_shuffle_writer_test
========= Host Frame: rmm::device_buffer::~device_buffer()
[0x29e8e8] in velox_gpu_shuffle_writer_test
========= Host Frame: cudf::detail::(anonymous
namespace)::batch_count_set_bits(cudf::host_span<unsigned int const* const,
18446744073709551615ul>, int, int, rmm::cuda_stream_view) [clone .cold]
[0x301bc4] in libcudf.so
========= Host Frame: cudf::detail::count_set_bits(unsigned int
const*, int, int, rmm::cuda_stream_view) [0xa3b210] in libcudf.so
========= Host Frame: cudf::null_count(unsigned int const*, int,
int, rmm::cuda_stream_view) [0xa3b57e] in libcudf.so
========= Host Frame: std::unique_ptr<cudf::column,
std::default_delete<cudf::column> > gluten::(anonymous
namespace)::DispatchColumn::readFlatColumn<(facebook::velox::TypeKind)5,
float>(cudf::type_id) [0x271e542] in libvelox.so
========= Host Frame: gluten::GpuBufferBatchResizer::next()
[0x271f086] in libvelox.so
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::shuffleWriteReadMultiBlocks(gluten::VeloxShuffleWriter&,
int, std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > > const&)
[0x12eeae] in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::testShuffleRoundTrip(gluten::VeloxShuffleWriter&,
std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >, int,
std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > >) [0x130786]
in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuHashPartitioningShuffleWriterTest_hashPart1Vector_Test::TestBody()
[0x1016ce] in velox_gpu_shuffle_writer_test
========= Host Frame: void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*) [0x29bbcc] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::Test::Run() [clone .part.0]
[0x29273b] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestInfo::Run() [0x292c51] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestSuite::Run() [clone .part.0]
[0x293160] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::internal::UnitTestImpl::RunAllTests()
[0x294320] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::UnitTest::Run() [0x2949df] in
velox_gpu_shuffle_writer_test
========= Host Frame: main [0xee986] in velox_gpu_shuffle_writer_test
=========
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified
launch failure" on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: void
cuda::__4::__override_fn_<rmm::mr::cuda_memory_resource, &(decltype
(({parm#1}.deallocate)({parm#2}, {parm#3}, {parm#4}, {parm#5}))
cuda::mr::__4::__deallocate_async<rmm::mr::cuda_memory_resource>(rmm::mr::cuda_memory_resource&,
cuda::__4::stream_ref, void*, unsigned long, unsigned long)), void, false,
false, cuda::__4::stream_ref, void*, unsigned long, unsigned
long>(cuda::std::__4::conditional<false, void const, void>::type*,
cuda::__4::stream_ref, void*, unsigned long, unsigned long) [0x271f8cb] in
libvelox.so
========= Host Frame: rmm::device_buffer::deallocate_async()
[0x29e7ea] in velox_gpu_shuffle_writer_test
========= Host Frame: rmm::device_buffer::~device_buffer()
[0x29e8e8] in velox_gpu_shuffle_writer_test
========= Host Frame: std::unique_ptr<rmm::device_buffer,
std::default_delete<rmm::device_buffer> >::~unique_ptr() [0x2720130] in
libvelox.so
========= Host Frame: std::unique_ptr<cudf::column,
std::default_delete<cudf::column> > gluten::(anonymous
namespace)::DispatchColumn::readFlatColumn<(facebook::velox::TypeKind)0,
bool>(cudf::type_id) [clone .cold] [0x2171366] in libvelox.so
========= Host Frame: gluten::GpuBufferBatchResizer::next()
[0x271f086] in libvelox.so
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::shuffleWriteReadMultiBlocks(gluten::VeloxShuffleWriter&,
int, std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > > const&)
[0x12eeae] in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::testShuffleRoundTrip(gluten::VeloxShuffleWriter&,
std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >, int,
std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > >) [0x130786]
in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuHashPartitioningShuffleWriterTest_hashPart1Vector_Test::TestBody()
[0x1016ce] in velox_gpu_shuffle_writer_test
========= Host Frame: void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*) [0x29bbcc] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::Test::Run() [clone .part.0]
[0x29273b] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestInfo::Run() [0x292c51] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestSuite::Run() [clone .part.0]
[0x293160] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::internal::UnitTestImpl::RunAllTests()
[0x294320] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::UnitTest::Run() [0x2949df] in
velox_gpu_shuffle_writer_test
========= Host Frame: main [0xee986] in velox_gpu_shuffle_writer_test
=========
========= Program hit cudaErrorLaunchFailure (error 719) due to "unspecified
launch failure" on CUDA API call to cudaFree.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: void
cuda::__4::__override_fn_<rmm::mr::cuda_memory_resource, &(decltype
(({parm#1}.deallocate)({parm#2}, {parm#3}, {parm#4}, {parm#5}))
cuda::mr::__4::__deallocate_async<rmm::mr::cuda_memory_resource>(rmm::mr::cuda_memory_resource&,
cuda::__4::stream_ref, void*, unsigned long, unsigned long)), void, false,
false, cuda::__4::stream_ref, void*, unsigned long, unsigned
long>(cuda::std::__4::conditional<false, void const, void>::type*,
cuda::__4::stream_ref, void*, unsigned long, unsigned long) [0x271f8cb] in
libvelox.so
========= Host Frame: rmm::device_buffer::deallocate_async()
[0x29e7ea] in velox_gpu_shuffle_writer_test
========= Host Frame: rmm::device_buffer::~device_buffer()
[0x29e8e8] in velox_gpu_shuffle_writer_test
========= Host Frame: std::unique_ptr<cudf::column,
std::default_delete<cudf::column> > gluten::(anonymous
namespace)::DispatchColumn::readFlatColumn<(facebook::velox::TypeKind)0,
bool>(cudf::type_id) [clone .cold] [0x21712fd] in libvelox.so
========= Host Frame: gluten::GpuBufferBatchResizer::next()
[0x271f086] in libvelox.so
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::shuffleWriteReadMultiBlocks(gluten::VeloxShuffleWriter&,
int, std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > > const&)
[0x12eeae] in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuVeloxShuffleWriterTest::testShuffleRoundTrip(gluten::VeloxShuffleWriter&,
std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >, int,
std::vector<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > >,
std::allocator<std::vector<std::shared_ptr<facebook::velox::RowVector>,
std::allocator<std::shared_ptr<facebook::velox::RowVector> > > > >) [0x130786]
in velox_gpu_shuffle_writer_test
========= Host Frame:
gluten::GpuHashPartitioningShuffleWriterTest_hashPart1Vector_Test::TestBody()
[0x1016ce] in velox_gpu_shuffle_writer_test
========= Host Frame: void
testing::internal::HandleExceptionsInMethodIfSupported<testing::Test,
void>(testing::Test*, void (testing::Test::*)(), char const*) [0x29bbcc] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::Test::Run() [clone .part.0]
[0x29273b] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestInfo::Run() [0x292c51] in
velox_gpu_shuffle_writer_test
========= Host Frame: testing::TestSuite::Run() [clone .part.0]
[0x293160] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::internal::UnitTestImpl::RunAllTests()
[0x294320] in velox_gpu_shuffle_writer_test
========= Host Frame: testing::UnitTest::Run() [0x2949df] in
velox_gpu_shuffle_writer_test
========= Host Frame: main [0xee986] in velox_gpu_shuffle_writer_test
=========
unknown file: Failure
C++ exception with description "CUDA error at:
/root/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/rmm-src/cpp/src/cuda_stream_view.cpp:45:
cudaErrorLaunchFailure unspecified launch failure" thrown in the test body.
[ FAILED ]
HashPartitioningShuffleWriterGroup/GpuHashPartitioningShuffleWriterTest.hashPart1Vector/0,
where GetParam() = 40-byte object <03-00 00-00 00-00 00-00 00-00 00-00 FF-FF
FF-FF 00-00 00-00 00-00 00-00 00-72 6B-00 73-74 73-00 00-00 00-00 00-00 00-00>
(1181 ms)
[----------] 1 test from
HashPartitioningShuffleWriterGroup/GpuHashPartitioningShuffleWriterTest (1181
ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1215 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ]
HashPartitioningShuffleWriterGroup/GpuHashPartitioningShuffleWriterTest.hashPart1Vector/0,
where GetParam() = 40-byte object <03-00 00-00 00-00 00-00 00-00 00-00 FF-FF
FF-FF 00-00 00-00 00-00 00-00 00-72 6B-00 73-74 73-00 00-00 00-00 00-00 00-00>
1 FAILED TEST
========= Target application returned an error
========= ERROR SUMMARY: 7 errors
```
**Root cause and the fix**
Currently the arrow null buffer point is passed to cudf::null_count(). The
fix is to pass the device buffer pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
