ihnorton commented on issue #42154: URL: https://github.com/apache/arrow/issues/42154#issuecomment-2173608139
The specific question here is if/when will arrow wheels update to AWS SDK 1.11? The reason for the question is to understand whether the mitigation for the issue described below will be available "soon", or we need to work around it (rename symbols, further patch the AWS SDK, etc?). For more background on the issue: - When pyarrow and tiledb-py are installed **from PyPI**, making S3 requests (via the AWS SDK) causes an abort on some platforms. - Standalone repro: https://gist.github.com/ihnorton/790b575944a5d09674e86a10700f1dab - this is a macOS-arm64 system - note that conda is used _only_ for python interpreter installation - the repro works in both directions: [import tiledb first -> pyarrow s3 request crashes](https://gist.github.com/ihnorton/790b575944a5d09674e86a10700f1dab#file-arrow-42154-a-txt) --- or --- [import pyarrow first -> tiledb s3 request crashes](https://gist.github.com/ihnorton/790b575944a5d09674e86a10700f1dab#file-arrow-42154-txt) - The top of the backtrace is: ``` 1 libarrow.1601.dylib 0x000000010e08b8d0 aws_fatal_assert + 80 2 libarrow.1601.dylib 0x000000010e08ab98 aws_mem_acquire + 64 3 libarrow.1601.dylib 0x000000010e09dd68 aws_string_new_from_cursor + 76 4 libarrow.1601.dylib 0x000000010e0975e4 aws_json_value_get_from_object + 44 5 libarrow.1601.dylib 0x000000010e083970 aws_endpoints_ruleset_new_from_string + 120 6 libarrow.1601.dylib 0x000000010e01d5a4 _ZN3Aws3Crt9Endpoints10RuleEngineC2ERK15aws_byte_cursorS5_P13aws_allocator + 48 7 libarrow.1601.dylib 0x000000010ddf7180 _ZN3Aws8Endpoint23DefaultEndpointProviderINS_2S321S3ClientConfigurationENS2_8Endpoint19S3BuiltInParametersENS4_25S3ClientContextParametersEEC2EPKcm + 116 8 libtiledb.dylib 0x0000000162bf367c _ZN3Aws2S38S3ClientC2ERKNS_6Client19ClientConfigurationENS2_15AWSAuthV4Signer20PayloadSigningPolicyEbNS0_34US_EAST_1_REGIONAL_ENDPOINT_OPTIONE + 980 9 libtiledb.dylib 0x000000016241a4bc _ZN6tiledb6common11make_sharedINS_2sm14TileDBS3ClientELi66EJRKNS2_12S3ParametersERN3Aws6Client19ClientConfigurationENS8_15AWSAuthV4Signer20PayloadSigningPolicyERKbEEENSt3__110shared_ptrIT_EERAT0__KcDpOT1_ + 92 10 libtiledb.dylib 0x0000000162403624 _ZNK6tiledb2sm2S311init_clientEv + 3428 ``` - This pre-existing arrow issue describes the underlying cause of the stack trace above: https://github.com/apache/arrow/issues/40262 - There is extensive prior discussion elsewhere on the AWS SDK repro, including https://github.com/aws/aws-sdk-cpp/issues/2699 - This is unrelated to conda, where -- as @h-vetinari notes -- the AWS SDK version is up-to-date; additionally, there is only a single copy of the SDK in the process image when both are installed from conda-forge. Summarizing: AWS has released a mitigation for the abort, implemented here: https://github.com/aws/aws-sdk-cpp/pull/2710. The mitigation is available in AWS SDK 1.11. TileDB wheels have updated to AWS SDK 1.11, but AFAICT all packages need to be updated for the mitigation to work. This issue will likely impact _any_ other library that bundles the AWS SDK in a wheel and is loaded at the same time as pyarrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
