Re: Trying to build to build pyarrow for python 2.7

[simba nyatsanga]

Hi Wes,

Great, thanks for the information.

On Tue, 16 Jan 2018 at 20:19 Wes McKinney  wrote:

> hi Simba -- the PyPI / pip wheels will only be updated when there is a
> new release. We'll either make a 0.8.1 release or 0.9.0 sometime in
> February depending on how development is progressing.
>
> - Wes
>
> On Sun, Jan 14, 2018 at 9:19 AM, simba nyatsanga 
> wrote:
> > Thanks a lot. I see that there's a PR  that's been opened to resolve the
> > encoding issue - https://github.com/apache/arrow/pull/1476
> >
> > Do you think this PR (if merged ) will also roll out as part of version
> > 0.9.0, or I'll be able to pip install with the merge commit as soon as
> it's
> > merged?
> >
> > Kind Regards
> >
> > On Sun, 14 Jan 2018 at 15:50 Uwe L. Korn  wrote:
> >
> >> Nice to hear that it worked.
> >>
> >> Updating the docs should not be necessary, we should rather see that we
> >> soon get a 0.9.0 release out (but that will also take some more weeks)
> >>
> >> Uwe
> >>
> >> On Sun, Jan 14, 2018, at 2:42 PM, simba nyatsanga wrote:
> >> > Amazing, thanks Uwe!
> >> >
> >> > I was able to build pyarrow successfully for python 2.7 using your
> >> > workaround. I appreciate that you've got a possible solution for the
> too.
> >> >
> >> > Besides the PR getting reviewed by more experienced maintainers, I'm
> >> > thinking to pull your branch and try the building process from
> scratch.
> >> > Otherwise I was wondering if it's valuable, in the meantime, to update
> >> the
> >> > docs with your work around?
> >> >
> >> > Kind Regards
> >> > Simba
> >> >
> >> > On Sun, 14 Jan 2018 at 15:17 Uwe L. Korn  wrote:
> >> >
> >> > > Hello Simba,
> >> > >
> >> > > it looks like you are running to
> >> > > https://issues.apache.org/jira/browse/ARROW-1856.
> >> > >
> >> > > To work around this issue, please "unset PARQUET_HOME" before you
> call
> >> the
> >> > > setup.py. Also set PKG_CONFIG_PATH, in your case this should be
> "export
> >> > >
> PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig".
> >> By
> >> > > doing this, you do the package discovery using pkg-config instead of
> >> the
> >> > > *_HOME variables. Currently this is the only path on which we can
> >> > > auto-detect the extension of the parquet shared library.
> >> > >
> >> > > Nevertheless, I will take a shot at fixing the issues as it seems
> that
> >> > > multiple users run into it.
> >> > >
> >> > > Uwe
> >> > >
> >> > > On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote:
> >> > > > Hi Wes,
> >> > > >
> >> > > > Apologies for the ambiguity there. To clarify, I used the conda
> >> > > > instructions only to create a conda environment. So I did this
> >> > > >
> >> > > > conda create -y -q -n pyarrow-dev \
> >> > > >   python=2.7 numpy six setuptools cython pandas pytest \
> >> > > >   cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy
> zlib \
> >> > > >   gflags brotli jemalloc lz4-c zstd -c conda-forge
> >> > > >
> >> > > >
> >> > > > I followed the instructions closely and I've stumbled upon a
> >> different
> >> > > > error from the one I initially had encountered. Now the issue
> seems
> >> to be
> >> > > > that when I'm building the Arrow C++ i.e running the following
> steps:
> >> > > >
> >> > > > mkdir parquet-cpp/build
> >> > > > pushd parquet-cpp/build
> >> > > >
> >> > > > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
> >> > > >   -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
> >> > > >   -DPARQUET_BUILD_BENCHMARKS=off \
> >> > > >   -DPARQUET_BUILD_EXECUTABLES=off \
> >> > > >   -DPARQUET_BUILD_TESTS=off \
> >> > > >   ..
> >> > > >
> >> > > > make -j4
> >> > > > make install
> >> > > > popd
> >> > > >
> >> > > >
> >> > > > The make install step generates *libparquet.1.3.2.dylib* as one of
> >> the
> >> > > > artefacts, as illustrated below:
> >> > > >
> >> > > > -- Install configuration: "RELEASE"-- Installing:
> >> > > >
> >> /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-
> >> > > > cppConfig.cmake--
> >> > > > Installing:
> /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/
> >> > > > cmake/parquet-cppConfigVersion.cmake--
> >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> >> > > > 1.3.2.dylib--
> >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> >> > > > 1.dylib--
> >> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/
> >> > > > libparquet.dylib--
> >> > > > Installing:
> /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
> >> > > > Installing:
> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> >> > > > column_reader.h--
> >> > > > Installing:
> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> >> > > > column_page.h--
> >> > > > Installing:
> /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> >> > > > column_scanner.h--
> >> > > > Installing:
> 

Re: Trying to build to build pyarrow for python 2.7

[simba nyatsanga]

Thanks a lot. I see that there's a PR  that's been opened to resolve the
encoding issue - https://github.com/apache/arrow/pull/1476

Do you think this PR (if merged ) will also roll out as part of version
0.9.0, or I'll be able to pip install with the merge commit as soon as it's
merged?

Kind Regards

On Sun, 14 Jan 2018 at 15:50 Uwe L. Korn  wrote:

> Nice to hear that it worked.
>
> Updating the docs should not be necessary, we should rather see that we
> soon get a 0.9.0 release out (but that will also take some more weeks)
>
> Uwe
>
> On Sun, Jan 14, 2018, at 2:42 PM, simba nyatsanga wrote:
> > Amazing, thanks Uwe!
> >
> > I was able to build pyarrow successfully for python 2.7 using your
> > workaround. I appreciate that you've got a possible solution for the too.
> >
> > Besides the PR getting reviewed by more experienced maintainers, I'm
> > thinking to pull your branch and try the building process from scratch.
> > Otherwise I was wondering if it's valuable, in the meantime, to update
> the
> > docs with your work around?
> >
> > Kind Regards
> > Simba
> >
> > On Sun, 14 Jan 2018 at 15:17 Uwe L. Korn  wrote:
> >
> > > Hello Simba,
> > >
> > > it looks like you are running to
> > > https://issues.apache.org/jira/browse/ARROW-1856.
> > >
> > > To work around this issue, please "unset PARQUET_HOME" before you call
> the
> > > setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export
> > > PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig".
> By
> > > doing this, you do the package discovery using pkg-config instead of
> the
> > > *_HOME variables. Currently this is the only path on which we can
> > > auto-detect the extension of the parquet shared library.
> > >
> > > Nevertheless, I will take a shot at fixing the issues as it seems that
> > > multiple users run into it.
> > >
> > > Uwe
> > >
> > > On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote:
> > > > Hi Wes,
> > > >
> > > > Apologies for the ambiguity there. To clarify, I used the conda
> > > > instructions only to create a conda environment. So I did this
> > > >
> > > > conda create -y -q -n pyarrow-dev \
> > > >   python=2.7 numpy six setuptools cython pandas pytest \
> > > >   cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
> > > >   gflags brotli jemalloc lz4-c zstd -c conda-forge
> > > >
> > > >
> > > > I followed the instructions closely and I've stumbled upon a
> different
> > > > error from the one I initially had encountered. Now the issue seems
> to be
> > > > that when I'm building the Arrow C++ i.e running the following steps:
> > > >
> > > > mkdir parquet-cpp/build
> > > > pushd parquet-cpp/build
> > > >
> > > > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
> > > >   -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
> > > >   -DPARQUET_BUILD_BENCHMARKS=off \
> > > >   -DPARQUET_BUILD_EXECUTABLES=off \
> > > >   -DPARQUET_BUILD_TESTS=off \
> > > >   ..
> > > >
> > > > make -j4
> > > > make install
> > > > popd
> > > >
> > > >
> > > > The make install step generates *libparquet.1.3.2.dylib* as one of
> the
> > > > artefacts, as illustrated below:
> > > >
> > > > -- Install configuration: "RELEASE"-- Installing:
> > > >
> /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-
> > > > cppConfig.cmake--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/
> > > > cmake/parquet-cppConfigVersion.cmake--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > > > 1.3.2.dylib--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > > > 1.dylib--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/
> > > > libparquet.dylib--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > column_reader.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > column_page.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > column_scanner.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > column_writer.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > encoding.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > exception.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > file_reader.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > file_writer.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > metadata.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > printer.h--
> > > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > > properties.h--
> > > > Installing: 

Re: Trying to build to build pyarrow for python 2.7

[Uwe L. Korn]

Nice to hear that it worked.

Updating the docs should not be necessary, we should rather see that we soon 
get a 0.9.0 release out (but that will also take some more weeks)

Uwe

On Sun, Jan 14, 2018, at 2:42 PM, simba nyatsanga wrote:
> Amazing, thanks Uwe!
> 
> I was able to build pyarrow successfully for python 2.7 using your
> workaround. I appreciate that you've got a possible solution for the too.
> 
> Besides the PR getting reviewed by more experienced maintainers, I'm
> thinking to pull your branch and try the building process from scratch.
> Otherwise I was wondering if it's valuable, in the meantime, to update the
> docs with your work around?
> 
> Kind Regards
> Simba
> 
> On Sun, 14 Jan 2018 at 15:17 Uwe L. Korn  wrote:
> 
> > Hello Simba,
> >
> > it looks like you are running to
> > https://issues.apache.org/jira/browse/ARROW-1856.
> >
> > To work around this issue, please "unset PARQUET_HOME" before you call the
> > setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export
> > PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig". By
> > doing this, you do the package discovery using pkg-config instead of the
> > *_HOME variables. Currently this is the only path on which we can
> > auto-detect the extension of the parquet shared library.
> >
> > Nevertheless, I will take a shot at fixing the issues as it seems that
> > multiple users run into it.
> >
> > Uwe
> >
> > On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote:
> > > Hi Wes,
> > >
> > > Apologies for the ambiguity there. To clarify, I used the conda
> > > instructions only to create a conda environment. So I did this
> > >
> > > conda create -y -q -n pyarrow-dev \
> > >   python=2.7 numpy six setuptools cython pandas pytest \
> > >   cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
> > >   gflags brotli jemalloc lz4-c zstd -c conda-forge
> > >
> > >
> > > I followed the instructions closely and I've stumbled upon a different
> > > error from the one I initially had encountered. Now the issue seems to be
> > > that when I'm building the Arrow C++ i.e running the following steps:
> > >
> > > mkdir parquet-cpp/build
> > > pushd parquet-cpp/build
> > >
> > > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
> > >   -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
> > >   -DPARQUET_BUILD_BENCHMARKS=off \
> > >   -DPARQUET_BUILD_EXECUTABLES=off \
> > >   -DPARQUET_BUILD_TESTS=off \
> > >   ..
> > >
> > > make -j4
> > > make install
> > > popd
> > >
> > >
> > > The make install step generates *libparquet.1.3.2.dylib* as one of the
> > > artefacts, as illustrated below:
> > >
> > > -- Install configuration: "RELEASE"-- Installing:
> > > /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-
> > > cppConfig.cmake--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/
> > > cmake/parquet-cppConfigVersion.cmake--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > > 1.3.2.dylib--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > > 1.dylib--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/
> > > libparquet.dylib--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > column_reader.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > column_page.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > column_scanner.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > column_writer.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > encoding.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > exception.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > file_reader.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > file_writer.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > metadata.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > printer.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > properties.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > schema.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > statistics.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > types.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > > parquet_version.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/
> > > parquet.pc--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > > io.h--
> > > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > > reader.h--
> 

Re: Trying to build to build pyarrow for python 2.7

[simba nyatsanga]

Amazing, thanks Uwe!

I was able to build pyarrow successfully for python 2.7 using your
workaround. I appreciate that you've got a possible solution for the too.

Besides the PR getting reviewed by more experienced maintainers, I'm
thinking to pull your branch and try the building process from scratch.
Otherwise I was wondering if it's valuable, in the meantime, to update the
docs with your work around?

Kind Regards
Simba

On Sun, 14 Jan 2018 at 15:17 Uwe L. Korn  wrote:

> Hello Simba,
>
> it looks like you are running to
> https://issues.apache.org/jira/browse/ARROW-1856.
>
> To work around this issue, please "unset PARQUET_HOME" before you call the
> setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export
> PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig". By
> doing this, you do the package discovery using pkg-config instead of the
> *_HOME variables. Currently this is the only path on which we can
> auto-detect the extension of the parquet shared library.
>
> Nevertheless, I will take a shot at fixing the issues as it seems that
> multiple users run into it.
>
> Uwe
>
> On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote:
> > Hi Wes,
> >
> > Apologies for the ambiguity there. To clarify, I used the conda
> > instructions only to create a conda environment. So I did this
> >
> > conda create -y -q -n pyarrow-dev \
> >   python=2.7 numpy six setuptools cython pandas pytest \
> >   cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
> >   gflags brotli jemalloc lz4-c zstd -c conda-forge
> >
> >
> > I followed the instructions closely and I've stumbled upon a different
> > error from the one I initially had encountered. Now the issue seems to be
> > that when I'm building the Arrow C++ i.e running the following steps:
> >
> > mkdir parquet-cpp/build
> > pushd parquet-cpp/build
> >
> > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
> >   -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
> >   -DPARQUET_BUILD_BENCHMARKS=off \
> >   -DPARQUET_BUILD_EXECUTABLES=off \
> >   -DPARQUET_BUILD_TESTS=off \
> >   ..
> >
> > make -j4
> > make install
> > popd
> >
> >
> > The make install step generates *libparquet.1.3.2.dylib* as one of the
> > artefacts, as illustrated below:
> >
> > -- Install configuration: "RELEASE"-- Installing:
> > /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-
> > cppConfig.cmake--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/
> > cmake/parquet-cppConfigVersion.cmake--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > 1.3.2.dylib--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > 1.dylib--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/
> > libparquet.dylib--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_page.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_scanner.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_writer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > encoding.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > exception.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > file_reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > file_writer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > metadata.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > printer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > properties.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > schema.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > statistics.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > types.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > parquet_version.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/
> > parquet.pc--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > io.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > writer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > schema.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > arrow/reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > arrow/schema.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > arrow/writer.h--
> > 

Re: Trying to build to build pyarrow for python 2.7

[Uwe L. Korn]

Just pushed a possible fix for this: https://github.com/apache/arrow/pull/1477

On Sun, Jan 14, 2018, at 2:17 PM, Uwe L. Korn wrote:
> Hello Simba,
> 
> it looks like you are running to 
> https://issues.apache.org/jira/browse/ARROW-1856.
> 
> To work around this issue, please "unset PARQUET_HOME" before you call 
> the setup.py. Also set PKG_CONFIG_PATH, in your case this should be 
> "export PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/
> pkgconfig". By doing this, you do the package discovery using pkg-config 
> instead of the *_HOME variables. Currently this is the only path on 
> which we can auto-detect the extension of the parquet shared library.
> 
> Nevertheless, I will take a shot at fixing the issues as it seems that 
> multiple users run into it.
> 
> Uwe
> 
> On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote:
> > Hi Wes,
> > 
> > Apologies for the ambiguity there. To clarify, I used the conda
> > instructions only to create a conda environment. So I did this
> > 
> > conda create -y -q -n pyarrow-dev \
> >   python=2.7 numpy six setuptools cython pandas pytest \
> >   cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
> >   gflags brotli jemalloc lz4-c zstd -c conda-forge
> > 
> > 
> > I followed the instructions closely and I've stumbled upon a different
> > error from the one I initially had encountered. Now the issue seems to be
> > that when I'm building the Arrow C++ i.e running the following steps:
> > 
> > mkdir parquet-cpp/build
> > pushd parquet-cpp/build
> > 
> > cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
> >   -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
> >   -DPARQUET_BUILD_BENCHMARKS=off \
> >   -DPARQUET_BUILD_EXECUTABLES=off \
> >   -DPARQUET_BUILD_TESTS=off \
> >   ..
> > 
> > make -j4
> > make install
> > popd
> > 
> > 
> > The make install step generates *libparquet.1.3.2.dylib* as one of the
> > artefacts, as illustrated below:
> > 
> > -- Install configuration: "RELEASE"-- Installing:
> > /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-
> > cppConfig.cmake--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/
> > cmake/parquet-cppConfigVersion.cmake--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > 1.3.2.dylib--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> > 1.dylib--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/
> > libparquet.dylib--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_page.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_scanner.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > column_writer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > encoding.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > exception.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > file_reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > file_writer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > metadata.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > printer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > properties.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > schema.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > statistics.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > types.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > parquet_version.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/
> > parquet.pc--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > io.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > writer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> > schema.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > arrow/reader.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > arrow/schema.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> > arrow/writer.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> > buffer-builder.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> > comparison.h--
> > Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> > logging.h--
> > Installing: 

Re: Trying to build to build pyarrow for python 2.7

[Uwe L. Korn]

Hello Simba,

it looks like you are running to 
https://issues.apache.org/jira/browse/ARROW-1856.

To work around this issue, please "unset PARQUET_HOME" before you call the 
setup.py. Also set PKG_CONFIG_PATH, in your case this should be "export 
PKG_CONFIG_PATH=/Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig". By doing 
this, you do the package discovery using pkg-config instead of the *_HOME 
variables. Currently this is the only path on which we can auto-detect the 
extension of the parquet shared library.

Nevertheless, I will take a shot at fixing the issues as it seems that multiple 
users run into it.

Uwe

On Thu, Jan 11, 2018, at 11:42 PM, simba nyatsanga wrote:
> Hi Wes,
> 
> Apologies for the ambiguity there. To clarify, I used the conda
> instructions only to create a conda environment. So I did this
> 
> conda create -y -q -n pyarrow-dev \
>   python=2.7 numpy six setuptools cython pandas pytest \
>   cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
>   gflags brotli jemalloc lz4-c zstd -c conda-forge
> 
> 
> I followed the instructions closely and I've stumbled upon a different
> error from the one I initially had encountered. Now the issue seems to be
> that when I'm building the Arrow C++ i.e running the following steps:
> 
> mkdir parquet-cpp/build
> pushd parquet-cpp/build
> 
> cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
>   -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
>   -DPARQUET_BUILD_BENCHMARKS=off \
>   -DPARQUET_BUILD_EXECUTABLES=off \
>   -DPARQUET_BUILD_TESTS=off \
>   ..
> 
> make -j4
> make install
> popd
> 
> 
> The make install step generates *libparquet.1.3.2.dylib* as one of the
> artefacts, as illustrated below:
> 
> -- Install configuration: "RELEASE"-- Installing:
> /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-
> cppConfig.cmake--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/
> cmake/parquet-cppConfigVersion.cmake--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> 1.3.2.dylib--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.
> 1.dylib--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/
> libparquet.dylib--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_page.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_scanner.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> column_writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> encoding.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> exception.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> file_reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> file_writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> metadata.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> printer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> properties.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> schema.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> statistics.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> types.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> parquet_version.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/
> parquet.pc--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> io.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/
> schema.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> arrow/reader.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> arrow/schema.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/
> arrow/writer.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> buffer-builder.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> comparison.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> logging.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> macros.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> memory.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> stopwatch.h--
> Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/
> visibility.h
> 
> 
> Subsequently when I want to build a standalone pyarrow 

Re: Trying to build to build pyarrow for python 2.7

[simba nyatsanga]

Hi Wes,

Apologies for the ambiguity there. To clarify, I used the conda
instructions only to create a conda environment. So I did this

conda create -y -q -n pyarrow-dev \
  python=2.7 numpy six setuptools cython pandas pytest \
  cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
  gflags brotli jemalloc lz4-c zstd -c conda-forge


I followed the instructions closely and I've stumbled upon a different
error from the one I initially had encountered. Now the issue seems to be
that when I'm building the Arrow C++ i.e running the following steps:

mkdir parquet-cpp/build
pushd parquet-cpp/build

cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
  -DCMAKE_INSTALL_PREFIX=$PARQUET_HOME \
  -DPARQUET_BUILD_BENCHMARKS=off \
  -DPARQUET_BUILD_EXECUTABLES=off \
  -DPARQUET_BUILD_TESTS=off \
  ..

make -j4
make install
popd


The make install step generates *libparquet.1.3.2.dylib* as one of the
artefacts, as illustrated below:

-- Install configuration: "RELEASE"-- Installing:
/Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-cppConfig.cmake--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/share/parquet-cpp/cmake/parquet-cppConfigVersion.cmake--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.1.3.2.dylib--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.1.dylib--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.dylib--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/libparquet.a--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/column_reader.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/column_page.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/column_scanner.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/column_writer.h--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/encoding.h--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/exception.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/file_reader.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/file_writer.h--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/metadata.h--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/printer.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/properties.h--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/schema.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/statistics.h--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/types.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/parquet_version.h--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/lib/pkgconfig/parquet.pc--
Installing: /Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/io.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/reader.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/writer.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/api/schema.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/arrow/reader.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/arrow/schema.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/arrow/writer.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/buffer-builder.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/comparison.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/logging.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/macros.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/memory.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/stopwatch.h--
Installing: 
/Users/simba/anaconda/envs/pyarrow-dev/include/parquet/util/visibility.h


Subsequently when I want to build a standalone pyarrow wheel by running
this step:

python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
   --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel



Then I get an error where one of the build steps  in the *CMakelists.txt*
expects to find *libparquet.1.0.0.dylib*. The  error is illustrated below:

running build_ext-- Runnning cmake for pyarrow
cmake  -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev/bin/python
 -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on
-DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release
/Users/simbarashenyatsanga/Projects/personal/oss/arrow/python
INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++
INFOCompiler version: Apple LLVM version 8.0.0
(clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0
Thread model: posixInstalledDir: /Library/Developer/CommandLineTools/usr/bin

INFOCompiler id: Clang
Selected compiler clang 3.8.0svn

Re: Trying to build to build pyarrow for python 2.7

[Wes McKinney]

hi Simba -- when you say "virtual env", I want to be sure you are
talking about a conda environment and not a virtualenv. Using these
together will cause all kinds of problems -- occasionally mixing pip
and conda installs can break conda environments, too.

The ones I linked to are the generated version of the development.rst
file (perhaps at a different revision, I'm working on updating the
website)

- Wes

On Thu, Jan 11, 2018 at 2:09 AM, simba nyatsanga  wrote:
> Hi Wes,
>
> Thanks for the response. I was following the development instructions on
> Github here:
> https://github.com/apache/arrow/blob/master/python/doc/source/development.rst
>
> I took MacOS option and installed my virtual env via conda. I must've
> missed an instruction when trying the 2.7 install, because I was able to
> successfully install for 3.6.
>
> Although it looks like the instructions on Github are similar to the ones
> you linked, I will give it another go with the later.
>
> Kind Regards
> Simba
>
> On Thu, 11 Jan 2018 at 00:51 Wes McKinney  wrote:
>
>> hi Simba,
>>
>> Are you following development instructions in
>>
>> http://arrow.apache.org/docs/python/development.html#developing-on-linux-and-macos
>> or something else?
>>
>> - Wes
>>
>> On Wed, Jan 10, 2018 at 11:20 AM, simba nyatsanga
>>  wrote:
>> > Hi,
>> >
>> > I've created a python 2.7 virtualenv in my attempt to build the pyarrow
>> > project. But I'm having trouble running one of commands as specified in
>> the
>> > development docs on Github, specifically this command:
>> >
>> > cd arrow/python
>> > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
>> >--with-parquet --with-plasma --inplace
>> >
>> > The error output looks like this:
>> >
>> > running build_ext-- Runnning cmake for pyarrow
>> > cmake
>> -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python
>> >  -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on
>> > -DCMAKE_BUILD_TYPE= /Users/simba/Projects/personal/oss/arrow/python
>> > INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++
>> > INFOCompiler version: Apple LLVM version 8.0.0
>> > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0
>> > Thread model: posixInstalledDir:
>> /Library/Developer/CommandLineTools/usr/bin
>> >
>> > INFOCompiler id: Clang
>> > Selected compiler clang 3.8.0svn
>> > Configured for DEBUG build (set with cmake
>> > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- Build
>> > output directory:
>> > /Users/simba/Projects/personal/oss/arrow/python/build/debug/--
>> > Checking for module 'arrow'--   No package 'arrow' found-- Found the
>> > Arrow core library:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib--
>> > Found the Arrow Python library:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib
>> > Added shared library dependency arrow:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib
>> > Added shared library dependency arrow_python:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib--
>> > Checking for module 'parquet'--   No package 'parquet' found-- Found
>> > the Parquet library:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib
>> > Added shared library dependency parquet:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib--
>> > Checking for module 'plasma'--   No package 'plasma' found-- Found the
>> > Plasma core library:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
>> > Found Plasma executable:
>> > Added shared library dependency libplasma:
>> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
>> > Configuring done-- Generating done-- Build files have been written to:
>> > /Users/simba/Projects/personal/oss/arrow/python-- Finished cmake for
>> > pyarrow-- Running cmake --build for pyarrow
>> > makemake: *** No targets specified and no makefile found.  Stop.error:
>> > command 'make' failed with exit status 2
>> >
>> >
>> > It looks like there's a change dir happening at this line in the
>> setup.py:
>> > https://github.com/apache/arrow/blob/master/python/setup.py#L136
>> > Which, in my case, is switching to the temp build which doesn't have the
>> > required Makefile to run the make command.
>> >
>> > I could be missing something because I was able to build the project
>> > successfully for python3. But I'd like to build it in python2.7 to
>> attempt
>> > a bug fix for this issue:
>> https://issues.apache.org/jira/browse/ARROW-1976
>> >
>> > Thanks for help.
>> >
>> > Kind Regards
>> > Simba
>>

Re: Trying to build to build pyarrow for python 2.7

[simba nyatsanga]

Hi Wes,

Thanks for the response. I was following the development instructions on
Github here:
https://github.com/apache/arrow/blob/master/python/doc/source/development.rst

I took MacOS option and installed my virtual env via conda. I must've
missed an instruction when trying the 2.7 install, because I was able to
successfully install for 3.6.

Although it looks like the instructions on Github are similar to the ones
you linked, I will give it another go with the later.

Kind Regards
Simba

On Thu, 11 Jan 2018 at 00:51 Wes McKinney  wrote:

> hi Simba,
>
> Are you following development instructions in
>
> http://arrow.apache.org/docs/python/development.html#developing-on-linux-and-macos
> or something else?
>
> - Wes
>
> On Wed, Jan 10, 2018 at 11:20 AM, simba nyatsanga
>  wrote:
> > Hi,
> >
> > I've created a python 2.7 virtualenv in my attempt to build the pyarrow
> > project. But I'm having trouble running one of commands as specified in
> the
> > development docs on Github, specifically this command:
> >
> > cd arrow/python
> > python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
> >--with-parquet --with-plasma --inplace
> >
> > The error output looks like this:
> >
> > running build_ext-- Runnning cmake for pyarrow
> > cmake
> -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python
> >  -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on
> > -DCMAKE_BUILD_TYPE= /Users/simba/Projects/personal/oss/arrow/python
> > INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++
> > INFOCompiler version: Apple LLVM version 8.0.0
> > (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0
> > Thread model: posixInstalledDir:
> /Library/Developer/CommandLineTools/usr/bin
> >
> > INFOCompiler id: Clang
> > Selected compiler clang 3.8.0svn
> > Configured for DEBUG build (set with cmake
> > -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- Build
> > output directory:
> > /Users/simba/Projects/personal/oss/arrow/python/build/debug/--
> > Checking for module 'arrow'--   No package 'arrow' found-- Found the
> > Arrow core library:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib--
> > Found the Arrow Python library:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib
> > Added shared library dependency arrow:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib
> > Added shared library dependency arrow_python:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib--
> > Checking for module 'parquet'--   No package 'parquet' found-- Found
> > the Parquet library:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib
> > Added shared library dependency parquet:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib--
> > Checking for module 'plasma'--   No package 'plasma' found-- Found the
> > Plasma core library:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
> > Found Plasma executable:
> > Added shared library dependency libplasma:
> > /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
> > Configuring done-- Generating done-- Build files have been written to:
> > /Users/simba/Projects/personal/oss/arrow/python-- Finished cmake for
> > pyarrow-- Running cmake --build for pyarrow
> > makemake: *** No targets specified and no makefile found.  Stop.error:
> > command 'make' failed with exit status 2
> >
> >
> > It looks like there's a change dir happening at this line in the
> setup.py:
> > https://github.com/apache/arrow/blob/master/python/setup.py#L136
> > Which, in my case, is switching to the temp build which doesn't have the
> > required Makefile to run the make command.
> >
> > I could be missing something because I was able to build the project
> > successfully for python3. But I'd like to build it in python2.7 to
> attempt
> > a bug fix for this issue:
> https://issues.apache.org/jira/browse/ARROW-1976
> >
> > Thanks for help.
> >
> > Kind Regards
> > Simba
>

Re: Trying to build to build pyarrow for python 2.7

[Wes McKinney]

hi Simba,

Are you following development instructions in
http://arrow.apache.org/docs/python/development.html#developing-on-linux-and-macos
or something else?

- Wes

On Wed, Jan 10, 2018 at 11:20 AM, simba nyatsanga
 wrote:
> Hi,
>
> I've created a python 2.7 virtualenv in my attempt to build the pyarrow
> project. But I'm having trouble running one of commands as specified in the
> development docs on Github, specifically this command:
>
> cd arrow/python
> python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
>--with-parquet --with-plasma --inplace
>
> The error output looks like this:
>
> running build_ext-- Runnning cmake for pyarrow
> cmake  
> -DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python
>  -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on
> -DCMAKE_BUILD_TYPE= /Users/simba/Projects/personal/oss/arrow/python
> INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++
> INFOCompiler version: Apple LLVM version 8.0.0
> (clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0
> Thread model: posixInstalledDir: /Library/Developer/CommandLineTools/usr/bin
>
> INFOCompiler id: Clang
> Selected compiler clang 3.8.0svn
> Configured for DEBUG build (set with cmake
> -DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- Build
> output directory:
> /Users/simba/Projects/personal/oss/arrow/python/build/debug/--
> Checking for module 'arrow'--   No package 'arrow' found-- Found the
> Arrow core library:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib--
> Found the Arrow Python library:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib
> Added shared library dependency arrow:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib
> Added shared library dependency arrow_python:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib--
> Checking for module 'parquet'--   No package 'parquet' found-- Found
> the Parquet library:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib
> Added shared library dependency parquet:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib--
> Checking for module 'plasma'--   No package 'plasma' found-- Found the
> Plasma core library:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
> Found Plasma executable:
> Added shared library dependency libplasma:
> /Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
> Configuring done-- Generating done-- Build files have been written to:
> /Users/simba/Projects/personal/oss/arrow/python-- Finished cmake for
> pyarrow-- Running cmake --build for pyarrow
> makemake: *** No targets specified and no makefile found.  Stop.error:
> command 'make' failed with exit status 2
>
>
> It looks like there's a change dir happening at this line in the setup.py:
> https://github.com/apache/arrow/blob/master/python/setup.py#L136
> Which, in my case, is switching to the temp build which doesn't have the
> required Makefile to run the make command.
>
> I could be missing something because I was able to build the project
> successfully for python3. But I'd like to build it in python2.7 to attempt
> a bug fix for this issue: https://issues.apache.org/jira/browse/ARROW-1976
>
> Thanks for help.
>
> Kind Regards
> Simba

Trying to build to build pyarrow for python 2.7

[simba nyatsanga]

Hi,

I've created a python 2.7 virtualenv in my attempt to build the pyarrow
project. But I'm having trouble running one of commands as specified in the
development docs on Github, specifically this command:

cd arrow/python
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE \
   --with-parquet --with-plasma --inplace

The error output looks like this:

running build_ext-- Runnning cmake for pyarrow
cmake  
-DPYTHON_EXECUTABLE=/Users/simba/anaconda/envs/pyarrow-dev-py2.7/bin/python
 -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=on
-DCMAKE_BUILD_TYPE= /Users/simba/Projects/personal/oss/arrow/python
INFOCompiler command: /Library/Developer/CommandLineTools/usr/bin/c++
INFOCompiler version: Apple LLVM version 8.0.0
(clang-800.0.42.1)Target: x86_64-apple-darwin15.6.0
Thread model: posixInstalledDir: /Library/Developer/CommandLineTools/usr/bin

INFOCompiler id: Clang
Selected compiler clang 3.8.0svn
Configured for DEBUG build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})-- Build Type: DEBUG-- Build
output directory:
/Users/simba/Projects/personal/oss/arrow/python/build/debug/--
Checking for module 'arrow'--   No package 'arrow' found-- Found the
Arrow core library:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib--
Found the Arrow Python library:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib
Added shared library dependency arrow:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow.dylib
Added shared library dependency arrow_python:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libarrow_python.dylib--
Checking for module 'parquet'--   No package 'parquet' found-- Found
the Parquet library:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib
Added shared library dependency parquet:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libparquet.dylib--
Checking for module 'plasma'--   No package 'plasma' found-- Found the
Plasma core library:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
Found Plasma executable:
Added shared library dependency libplasma:
/Users/simba/anaconda/envs/pyarrow-dev-py2.7/lib/libplasma.dylib--
Configuring done-- Generating done-- Build files have been written to:
/Users/simba/Projects/personal/oss/arrow/python-- Finished cmake for
pyarrow-- Running cmake --build for pyarrow
makemake: *** No targets specified and no makefile found.  Stop.error:
command 'make' failed with exit status 2


It looks like there's a change dir happening at this line in the setup.py:
https://github.com/apache/arrow/blob/master/python/setup.py#L136
Which, in my case, is switching to the temp build which doesn't have the
required Makefile to run the make command.

I could be missing something because I was able to build the project
successfully for python3. But I'd like to build it in python2.7 to attempt
a bug fix for this issue: https://issues.apache.org/jira/browse/ARROW-1976

Thanks for help.

Kind Regards
Simba