[
https://issues.apache.org/jira/browse/ARROW-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025745#comment-17025745
]
Michael Marino commented on ARROW-5158:
---------------------------------------
Note that this is critical for applications using pyarrow in environments where
space is limited. This is the case with e.g. ([AWS
Lambda|[https://docs.aws.amazon.com/lambda/latest/dg/limits.html]]), where the
package size of all modules needs to be below 250 MB. With this bug, it is
impossible to deploy pyarrow together with e.g. pandas/numpy, making it
impossible to load parquet files in lambda functions.
> [Packaging][Wheel] Symlink libraries in wheels
> ----------------------------------------------
>
> Key: ARROW-5158
> URL: https://issues.apache.org/jira/browse/ARROW-5158
> Project: Apache Arrow
> Issue Type: Bug
> Components: Packaging, Python
> Reporter: Krisztian Szucs
> Priority: Major
> Labels: wheel
>
> Libraries are copied instead of symlinking in linux and osx wheels, which
> result quiet big binaries:
>
> This is what the wheel contains before running auditwheel:
>
> {code}
> -rwxr-xr-x 1 root root 128K Apr 3 09:02 libarrow_boost_filesystem.so
> -rwxr-xr-x 1 root root 128K Apr 3 09:02 libarrow_boost_filesystem.so.1.66.0
> -rwxr-xr-x 1 root root 1.2M Apr 3 09:02 libarrow_boost_regex.so
> -rwxr-xr-x 1 root root 1.2M Apr 3 09:02 libarrow_boost_regex.so.1.66.0
> -rwxr-xr-x 1 root root 30K Apr 3 09:02 libarrow_boost_system.so
> -rwxr-xr-x 1 root root 30K Apr 3 09:02 libarrow_boost_system.so.1.66.0
> -rwxr-xr-x 1 root root 1.4M Apr 3 09:02 libarrow_python.so
> -rwxr-xr-x 1 root root 1.4M Apr 3 09:02 libarrow_python.so.14
> -rwxr-xr-x 1 root root 12M Apr 3 09:02 libarrow.so
> -rwxr-xr-x 1 root root 12M Apr 3 09:02 libarrow.so.14
> -rw-r--r-- 1 root root 6.1M Apr 3 09:02 lib.cpp
> -rwxr-xr-x 1 root root 2.4M Apr 3 09:02
> [lib.cpython-36m-x86_64-linux-gnu.so|http://lib.cpython-36m-x86_64-linux-gnu.so/]
> -rwxr-xr-x 1 root root 55M Apr 3 09:02 libgandiva.so
> -rwxr-xr-x 1 root root 55M Apr 3 09:02 libgandiva.so.14
> -rwxr-xr-x 1 root root 2.9M Apr 3 09:02 libparquet.so
> -rwxr-xr-x 1 root root 2.9M Apr 3 09:02 libparquet.so.14
> -rwxr-xr-x 1 root root 309K Apr 3 09:02 libplasma.so
> -rwxr-xr-x 1 root root 309K Apr 3 09:02 libplasma.so.14
> {code}
> After running auditwheel, the repaired wheel contains:
>
> {code}
> -rwxr-xr-x 1 root root 128K Apr 3 09:02 libarrow_boost_filesystem.so
> -rwxr-xr-x 1 root root 128K Apr 3 09:02 libarrow_boost_filesystem.so.1.66.0
> -rwxr-xr-x 1 root root 1.2M Apr 3 09:02 libarrow_boost_regex.so
> -rwxr-xr-x 1 root root 1.2M Apr 3 09:02 libarrow_boost_regex.so.1.66.0
> -rwxr-xr-x 1 root root 30K Apr 3 09:02 libarrow_boost_system.so
> -rwxr-xr-x 1 root root 30K Apr 3 09:02 libarrow_boost_system.so.1.66.0
> -rwxr-xr-x 1 root root 1.6M Apr 3 09:55 libarrow_python.so
> -rwxr-xr-x 1 root root 1.4M Apr 3 09:02 libarrow_python.so.14
> -rwxr-xr-x 1 root root 12M Apr 3 09:55 libarrow.so
> -rwxr-xr-x 1 root root 12M Apr 3 09:02 libarrow.so.14
> -rw-r--r-- 1 root root 6.1M Apr 3 09:02 lib.cpp
> -rwxr-xr-x 1 root root 2.5M Apr 3 09:55
> [lib.cpython-36m-x86_64-linux-gnu.so|http://lib.cpython-36m-x86_64-linux-gnu.so/]
> -rwxr-xr-x 1 root root 59M Apr 3 09:55 libgandiva.so
> -rwxr-xr-x 1 root root 55M Apr 3 09:02 libgandiva.so.14
> -rwxr-xr-x 1 root root 3.5M Apr 3 09:55 libparquet.so
> -rwxr-xr-x 1 root root 2.9M Apr 3 09:02 libparquet.so.14
> -rwxr-xr-x 1 root root 345K Apr 3 09:55 libplasma.so
> -rwxr-xr-x 1 root root 309K Apr 3 09:02 libplasma.so.14
> {code}
>
> Here is the output of auditwheel
> [https://travis-ci.org/kszucs/crossbow/builds/514605723#L3340]
> They should be symlinks, we have special code for this:
> https://github.com/apache/arrow/blob/4495305092411e8551c60341e273c8aa3c14b282/python/setup.py#L489-L499
> This is probably not going into the wheel as wheels are zip-files and they
> don't support symlinks by default. So we probably need to pass the
> `--symlinks` parameter to the wheel code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
