I have managed to successfully verify this RC today, confirming the Pandas
issue was the only thing blocking my vote. If we think the currently in
discussion is non-blocking, I'm happy to give my plus one vote.
Here are the verifications commands I ran (and runtimes), for others'
reference:
# Verify binaries other than wheels (took 2h 24m)
TEST_DEFAULT=0 TEST_BINARIES=1 TEST_WHEELS=0
./dev/release/verify-release-candidate.sh 12.0.0 0
# Test main languages and integrations (took 49m)
TEST_DEFAULT=0 TEST_CPP=1 TEST_JAVA=1 TEST_GO=1 TEST_JS=1 TEST_CSHARP=1
TEST_INTEGRATION=1 \
./dev/release/verify-release-candidate.sh 12.0.0 0
# Test Ruby (took 24m)
TEST_DEFAULT=0 TEST_RUBY=1 \
./dev/release/verify-release-candidate.sh 12.0.0 0
And then if I added the line
pip install 'pandas<2'
after verify-release-candidate.sh:726. I could successfully run:
# Test Python (took 23m)
TEST_DEFAULT=0 TEST_PYTHON=1 \
./dev/release/verify-release-candidate.sh 12.0.0 0
On Fri, Apr 28, 2023 at 8:59 AM Matthew Topol <m...@voltrondata.com.invalid>
wrote:
> Okay, I confirmed that by running the following two commands, the test
> failures that Jacob found on Manjarolinux were solved (at least in the
> container I was using)
>
> $ pacman -S libxcrypt-compat
> $ ln -s /usr/share/zoneinfo/America/New_York /etc/localtime
>
> For the second command it looks like the /etc/localtime symbolic link
> wasn't being set in the container and is leveraged by the Orc adapter
> tests. So setting the localtime (to any valid zone info) was sufficient to
> let the tests run and pass.
>
> Hope this helps!
>
>
> On Fri, Apr 28, 2023 at 11:49 AM Matthew Topol <m...@voltrondata.com>
> wrote:
>
> > Looks like this might be related:
> >
> https://unix.stackexchange.com/questions/691479/how-to-deal-with-missing-libcrypt-so-1-on-arch-linux
> > as manjaro also uses pacman and Arch Linux's packages.
> >
> > I'm re-running the verification right now after installing the
> recommended
> > package in that thread. I'll report back if it solves the issue.
> >
> > --Matt
> >
> > On Fri, Apr 28, 2023 at 11:29 AM Matthew Topol <m...@voltrondata.com>
> > wrote:
> >
> >> @Kou: I was able to reproduce the libcrypto failure that Jacob saw using
> >> https://hub.docker.com/r/manjarolinux/base though i did need to
> manually
> >> install git first since it doesn't come with it.
> >>
> >> $ pacman -Syu git
> >> $ git clone https://github.com/apache/arrow.git
> >> $ cd arrow
> >> $ TEST_DEFAULT=0 TEST_SOURCE=0 TEST_CPP=1 USE_CONDA=1
> >> dev/release/verify-release-candidate.sh 12.0.0 0
> >>
> >> That set of commands was sufficient to reproduce the error I believe (I
> >> did this on monday when I was poking around the failures but I
> definitely
> >> managed to see the same error pop up in a run). I'm running it again
> right
> >> now to confirm.
> >>
> >> --Matt
> >>
> >> On Thu, Apr 27, 2023 at 8:28 PM Sutou Kouhei <k...@clear-code.com>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> Thanks for sharing the log.
> >>>
> >>> libcrypto.so isn't related on the segmentation fault. It's
> >>> just for relating to showing backtrace.
> >>>
> >>> > perl: error while loading shared libraries: libcrypt.so.1:
> >>> > cannot open shared object file: No such file or directory
> >>>
> >>> This is happen at
> >>>
> >>>
> https://github.com/apache/arrow/blob/main/cpp/build-support/run-test.sh#L42
> >>> :
> >>>
> >>> TEST_NAME=$(echo $TEST_FILENAME | perl -pe 's/\..+?$//') # Remove
> path
> >>> and extension (if any).
> >>>
> >>> BTW, it seems that we should remove a Perl dependency from
> >>>
> https://github.com/apache/arrow/blob/main/cpp/build-support/run-test.sh
> >>> ...
> >>>
> >>>
> >>> I want to reproduce this problem on my environment. Could
> >>> you share your environment information? Did you use Manjaro
> >>> Linux this too?
> >>>
> >>>
> >>> Thanks,
> >>> --
> >>> kou
> >>>
> >>>
> >>> In <canva0dgkodpfde7_b8xuvmtkh5kdmzvmtpbofo82hqj17gu...@mail.gmail.com
> >
> >>> "Re: [VOTE] Release Apache Arrow 12.0.0 - RC0" on Thu, 27 Apr 2023
> >>> 23:54:58 +0200,
> >>> Jacob Wujciak <ja...@voltrondata.com.INVALID> wrote:
> >>>
> >>> > I have uploaded the log [1] for the run using conda with gandiva
> >>> active. It
> >>> > looks like there is an issue with libcrypt.so causing these tests to
> >>> > segfault.
> >>> >
> >>> > 1:
> https://gist.github.com/assignUser/cba0a13875de9d6a4f31000f585244f0
> >>> >
> >>> > On Thu, Apr 27, 2023 at 11:32 PM Will Jones <will.jones...@gmail.com
> >
> >>> wrote:
> >>> >
> >>> >> Hi Raul,
> >>> >>
> >>> >> It might be worth creating a new RC that fixes more of the test
> >>> issues,
> >>> >> even if they shouldn't be blockers. I've run the release script a
> few
> >>> >> different times, and after 1.5 hours (is that a normal runtime for
> >>> >> verification?) I get various test failures. So far the errors are in
> >>> the
> >>> >> TEST_PYTHON, TEST_WHEELS, and TEST_JAVA. I'll have more time
> tomorrow
> >>> to
> >>> >> look at the failures I am seeing.
> >>> >>
> >>> >> On Thu, Apr 27, 2023 at 8:09 AM Raúl Cumplido <
> raulcumpl...@gmail.com
> >>> >
> >>> >> wrote:
> >>> >>
> >>> >> > Hi,
> >>> >> >
> >>> >> > The vote for the RC has been open for 5 days.
> >>> >> >
> >>> >> > I will wait until tomorrow, if no more +1 votes are casted I
> >>> >> > understand that the issue related to the pandas failures
> >>> >> > (https://github.com/apache/arrow/issues/35321) is causing
> >>> verification
> >>> >> > to fail and we require a new RC with the above fix.
> >>> >> >
> >>> >> > Let me know if there are other blockers that should be included in
> >>> that
> >>> >> > case.
> >>> >> >
> >>> >> > Kind regards,
> >>> >> > Raúl
> >>> >> >
> >>> >> > El jue, 27 abr 2023 a las 16:08, Sutou Kouhei (<
> k...@clear-code.com
> >>> >)
> >>> >> > escribió:
> >>> >> > >
> >>> >> > > Hi,
> >>> >> > >
> >>> >> > > I tried this on a manjarolinux/base Docker image.
> >>> >> > >
> >>> >> > > I think that this is a problem of the Arch Linux's llvm
> >>> >> > > package. LLVMExports.cmake in the package doesn't provide
> >>> >> > > the LLVMX86CodeGen target:
> >>> >> > >
> >>> >> > > # grep add_library /usr/lib/cmake/llvm/LLVMExports.cmake
> >>> >> > > add_library(LLVMDemangle STATIC IMPORTED)
> >>> >> > > add_library(LLVMSupport STATIC IMPORTED)
> >>> >> > > add_library(LLVMTableGen STATIC IMPORTED)
> >>> >> > > add_library(LTO SHARED IMPORTED)
> >>> >> > > add_library(LLVMgold MODULE IMPORTED)
> >>> >> > > add_library(LLVM SHARED IMPORTED)
> >>> >> > > add_library(Remarks SHARED IMPORTED)
> >>> >> > >
> >>> >> > > FYI: LLVMExports.cmake on Debian GNU/Linux provides many
> >>> >> > > targets:
> >>> >> > >
> >>> >> > > $ grep add_library
> >>> /usr/lib/llvm-15/lib/cmake/llvm/LLVMExports.cmake |
> >>> >> > head
> >>> >> > > add_library(LLVMDemangle STATIC IMPORTED)
> >>> >> > > add_library(LLVMSupport STATIC IMPORTED)
> >>> >> > > add_library(LLVMTableGen STATIC IMPORTED)
> >>> >> > > add_library(LLVMTableGenGlobalISel STATIC IMPORTED)
> >>> >> > > add_library(LLVMCore STATIC IMPORTED)
> >>> >> > > add_library(LLVMFuzzerCLI STATIC IMPORTED)
> >>> >> > > add_library(LLVMFuzzMutate STATIC IMPORTED)
> >>> >> > > add_library(LLVMFileCheck STATIC IMPORTED)
> >>> >> > > add_library(LLVMInterfaceStub STATIC IMPORTED)
> >>> >> > > add_library(LLVMIRReader STATIC IMPORTED)
> >>> >> > >
> >>> >> > > $ grep add_library
> >>> /usr/lib/llvm-15/lib/cmake/llvm/LLVMExports.cmake |
> >>> >> > wc -l
> >>> >> > > 195
> >>> >> > >
> >>> >> > >
> >>> >> > > Thanks,
> >>> >> > > --
> >>> >> > > kou
> >>> >> > >
> >>> >> > > In <CANva0dgrkkaAb_dbnSLXuVxi=
> >>> h22i0yjhbtde-y3-rflox8...@mail.gmail.com
> >>> >> >
> >>> >> > > "Re: [VOTE] Release Apache Arrow 12.0.0 - RC0" on Tue, 25 Apr
> >>> 2023
> >>> >> > 23:50:21 +0200,
> >>> >> > > Jacob Wujciak <ja...@voltrondata.com.INVALID> wrote:
> >>> >> > >
> >>> >> > > > I checked out a trace for the cmake issue and LLVM 15.07 is
> >>> found
> >>> >> > > > correctly. The issue come from
> `llvm_map_components_to_libnames`
> >>> >> which
> >>> >> > > > complains about X86 not being in the lsit of libraries. But we
> >>> don't
> >>> >> > add
> >>> >> > > > that but rather it gets appended in the function?
> >>> >> > > >
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(228):
> >>> >> > > > get_property(LLVM_TARGETS_CONFIGURED GLOBAL PROPERTY
> >>> >> > > > LLVM_TARGETS_CONFIGURED )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(234): if(NOT
> >>> >> > LLVM_TARGETS_CONFIGURED
> >>> >> > > > )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(244): list(FIND
> >>> >> > > > LLVM_TARGETS_TO_BUILD X86 have_native_backend )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(245): list(FIND
> >>> >> link_components
> >>> >> > > > engine engine_required )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(246): if(NOT
> >>> engine_required
> >>> >> > EQUAL
> >>> >> > > > -1 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(255): list(FIND
> >>> >> link_components
> >>> >> > > > native native_required )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(256): if(NOT
> >>> native_required
> >>> >> > EQUAL
> >>> >> > > > -1 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(257): if(NOT
> >>> >> have_native_backend
> >>> >> > > > EQUAL -1 )
> >>> >> > > > */usr/lib/cmake/llvm/LLVM-Config.cmake(258): list(APPEND
> >>> >> > link_components
> >>> >> > > > X86 )*
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(263):
> >>> >> > > > llvm_expand_pseudo_components(link_components
> >>> >> > > >
> >>> >> >
> >>> >>
> >>>
> core;mcjit;native;ipo;bitreader;target;linker;analysis;debuginfodwarf;X86 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(127):
> set(link_components
> >>> >> > > >
> >>> >> >
> >>> >>
> >>>
> core;mcjit;native;ipo;bitreader;target;linker;analysis;debuginfodwarf;X86 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(128): foreach(c
> >>> >> > > >
> >>> >> >
> >>> >>
> >>>
> core;mcjit;native;ipo;bitreader;target;linker;analysis;debuginfodwarf;X86 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(130): list(FIND
> >>> >> > > > LLVM_TARGETS_TO_BUILD core idx )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(131): if(NOT idx LESS
> 0 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(159): elseif(c STREQUAL
> >>> >> > > > nativecodegen )
> >>> >> > > > [snip]
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(212): list(APPEND
> >>> >> > > > expanded_components debuginfodwarf )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(130): list(FIND
> >>> >> > > > LLVM_TARGETS_TO_BUILD X86 idx )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(131): if(NOT idx LESS
> 0 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(132): if(TARGET
> >>> >> LLVMX86CodeGen )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(134): else()
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(135): if(TARGET
> LLVMX86 )
> >>> >> > > > /usr/lib/cmake/llvm/LLVM-Config.cmake(137): else()
> >>> >> > > > */usr/lib/cmake/llvm/LLVM-Config.cmake(138):
> >>> message(FATAL_ERROR
> >>> >> > Target
> >>> >> > > > X86 is not in the set of libraries. )*
> >>> >> > > >
> >>> >> > > > On Tue, Apr 25, 2023 at 10:57 AM Raúl Cumplido <
> >>> >> raulcumpl...@gmail.com
> >>> >> > >
> >>> >> > > > wrote:
> >>> >> > > >
> >>> >> > > >> I have created the following issue for the new wheels test
> >>> failure
> >>> >> > > >> around pandas 2.0.1 :
> >>> https://github.com/apache/arrow/issues/35321
> >>> >> > > >>
> >>> >> > > >> I don't think we should create a new RC for that issue but
> I'm
> >>> happy
> >>> >> > > >> to know other people's thoughts around that.
> >>> >> > > >>
> >>> >> > > >> El lun, 24 abr 2023 a las 21:12, Raúl Cumplido
> >>> >> > > >> (<raulcumpl...@gmail.com>) escribió:
> >>> >> > > >> >
> >>> >> > > >> > El lun, 24 abr 2023 a las 18:53, Will Jones
> >>> >> > > >> > (<will.jones...@gmail.com>) escribió:
> >>> >> > > >> > >
> >>> >> > > >> > > I'm seeing failing Pandas tests in PyArrow when verifying
> >>> with
> >>> >> > > >> > >
> >>> >> > > >> > > USE_CONDA=1 dev/release/verify-release-candidate.sh
> 12.0.0
> >>> 0
> >>> >> > > >> > >
> >>> >> > > >> > >
> >>> >> > > >>
> >>> >> >
> >>> >>
> >>>
> pyarrow/tests/test_extension_type.py::test_extension_to_pandas_storage_type[registered_period_type0]
> >>> >> > > >> > > - NotImplementedError: extension<test.period<PeriodType>>
> >>> >> > > >> >
> >>> >> > > >> > This is also happening on our nightlies from today:
> >>> >> > > >> >
> >>> >> > > >>
> >>> >> >
> >>> >>
> >>>
> https://github.com/ursacomputing/crossbow/actions/runs/4786502455/jobs/8510514881
> >>> >> > > >> >
> >>> >> > > >> > There has been a new pandas release: 2.0.1 around 9 hours
> ago
> >>> >> which
> >>> >> > > >> > seems to be the causing issue:
> >>> >> > > >> > https://pypi.org/project/pandas/#history
> >>> >> > > >> >
> >>> >> > > >> > >
> >>> >> > > >> > > No one else is getting that?
> >>> >> > > >> > >
> >>> >> > > >> > >
> >>> >> > > >> > > On Sun, Apr 23, 2023 at 9:21 AM Raúl Cumplido <
> >>> >> > raulcumpl...@gmail.com>
> >>> >> > > >> > > wrote:
> >>> >> > > >> > >
> >>> >> > > >> > > > +1 (non binding)
> >>> >> > > >> > > >
> >>> >> > > >> > > > I have tested both SOURCES and BINARIES successfully
> >>> with:
> >>> >> > > >> > > > TEST_DEFAULT=0 TEST_SOURCE=1
> >>> >> > dev/release/verify-release-candidate.sh
> >>> >> > > >> > > > 12.0.0 0
> >>> >> > > >> > > > TEST_DEFAULT=0 TEST_WHEELS=1
> >>> >> > dev/release/verify-release-candidate.sh
> >>> >> > > >> > > > 12.0.0 0
> >>> >> > > >> > > > TEST_DEFAULT=0 TEST_BINARIES=1
> >>> >> > > >> dev/release/verify-release-candidate.sh
> >>> >> > > >> > > > 12.0.0 0
> >>> >> > > >> > > > with:
> >>> >> > > >> > > > * Python 3.10.6
> >>> >> > > >> > > > * gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
> >>> >> > > >> > > > * NVIDIA CUDA cuda_11.5.r11.5/compiler.30672275_0
> >>> >> > > >> > > > * openjdk version "17.0.6" 2023-01-17
> >>> >> > > >> > > > * ruby 3.0.2p107 (2021-07-07 revision 0db68f0233)
> >>> >> > > >> [x86_64-linux-gnu]
> >>> >> > > >> > > > * dotnet 7.0.203
> >>> >> > > >> > > > * Ubuntu 22.04 LTS
> >>> >> > > >> > > >
> >>> >> > > >> > > > El dom, 23 abr 2023 a las 12:59, Yibo Cai (<
> >>> yibo....@arm.com
> >>> >> >)
> >>> >> > > >> escribió:
> >>> >> > > >> > > > >
> >>> >> > > >> > > > > +1
> >>> >> > > >> > > > >
> >>> >> > > >> > > > > I ran the followings on Ubuntu-22.04, aarch64.
> >>> >> > > >> > > > >
> >>> >> > > >> > > > > TEST_DEFAULT=0 \
> >>> >> > > >> > > > > TEST_CPP=1 \
> >>> >> > > >> > > > > TEST_PYTHON=1 \
> >>> >> > > >> > > > > TEST_GO=1 \
> >>> >> > > >> > > > > dev/release/verify-release-candidate.sh 12.0.0 0
> >>> >> > > >> > > > >
> >>> >> > > >> > > > > TEST_DEFAULT=0 \
> >>> >> > > >> > > > > TEST_WHEELS=1 \
> >>> >> > > >> > > > > dev/release/verify-release-candidate.sh 12.0.0 0
> >>> >> > > >> > > > >
> >>> >> > > >> > > > >
> >>> >> > > >> > > > > On 4/23/23 14:40, Sutou Kouhei wrote:
> >>> >> > > >> > > > > > +1
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > I ran the followings on Debian GNU/Linux sid:
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > * TEST_DEFAULT=0 \
> >>> >> > > >> > > > > > TEST_SOURCE=1 \
> >>> >> > > >> > > > > > LANG=C \
> >>> >> > > >> > > > > > TZ=UTC \
> >>> >> > > >> > > > > > CUDAToolkit_ROOT=/usr \
> >>> >> > > >> > > > > >
> ARROW_CMAKE_OPTIONS="-DBoost_NO_BOOST_CMAKE=ON
> >>> >> > > >> > > > -Dxsimd_SOURCE=BUNDLED" \
> >>> >> > > >> > > > > > dev/release/verify-release-candidate.sh
> >>> 12.0.0 0
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > * TEST_DEFAULT=0 \
> >>> >> > > >> > > > > > TEST_APT=1 \
> >>> >> > > >> > > > > > LANG=C \
> >>> >> > > >> > > > > > dev/release/verify-release-candidate.sh
> >>> 12.0.0 0
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > * TEST_DEFAULT=0 \
> >>> >> > > >> > > > > > TEST_BINARY=1 \
> >>> >> > > >> > > > > > LANG=C \
> >>> >> > > >> > > > > > dev/release/verify-release-candidate.sh
> >>> 12.0.0 0
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > * TEST_DEFAULT=0 \
> >>> >> > > >> > > > > > TEST_JARS=1 \
> >>> >> > > >> > > > > > LANG=C \
> >>> >> > > >> > > > > > dev/release/verify-release-candidate.sh
> >>> 12.0.0 0
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > * TEST_DEFAULT=0 \
> >>> >> > > >> > > > > > TEST_PYTHON_VERSIONS=3.11 \
> >>> >> > > >> > > > > > TEST_WHEELS=1 \
> >>> >> > > >> > > > > > LANG=C \
> >>> >> > > >> > > > > > dev/release/verify-release-candidate.sh
> >>> 12.0.0 0
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > * TEST_DEFAULT=0 \
> >>> >> > > >> > > > > > TEST_YUM=1 \
> >>> >> > > >> > > > > > LANG=C \
> >>> >> > > >> > > > > > dev/release/verify-release-candidate.sh
> >>> 12.0.0 0
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > with:
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > * .NET SDK (6.0.406)
> >>> >> > > >> > > > > > * Python 3.11.2
> >>> >> > > >> > > > > > * gcc (Debian 12.2.0-14) 12.2.0
> >>> >> > > >> > > > > > * nvidia-cuda-dev 11.7.99~11.7.1-4
> >>> >> > > >> > > > > > * openjdk version "17.0.6" 2023-01-17
> >>> >> > > >> > > > > > * ruby 3.1.2p20 (2022-04-12 revision 4491bb740a)
> >>> >> > > >> [x86_64-linux-gnu]
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > >
> >>> >> > > >> > > > > > Thanks,
> >>> >> > > >> > > >
> >>> >> > > >>
> >>> >> >
> >>> >>
> >>>
> >>
>
