kou commented on code in PR #37821: URL: https://github.com/apache/arrow/pull/37821#discussion_r1335284897
########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten Review Comment: ```suggestion # Force some variables for Emscripten ``` ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + +# # override default in emscripten which is to not use shared libs +set_property(GLOBAL PROPERTY TARGET_SUPPORTS_SHARED_LIBS TRUE) Review Comment: Why do we need this? What shared libraries are used? ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) Review Comment: Can we do this in `build_zlib()` in `ThirdpartyToolchain.cmake`? ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + +# # override default in emscripten which is to not use shared libs +set_property(GLOBAL PROPERTY TARGET_SUPPORTS_SHARED_LIBS TRUE) + +# if we leave the system name as Emscripten, then it reloads the original Emscripten.cmake every time a project() command +# is run, which does bad things like disabling shared libraries +set(CMAKE_SYSTEM_NAME EmscriptenOverrides) + +set(CMAKE_C_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") +set(CMAKE_CXX_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") Review Comment: Why do we need `-sUSE_ZLIB=1`? Could you tell me the document URL for `SIDE_MODULE=1`? Why do we need to specify `-fPIC` explicitly? CMake's `POSITION_INDEPENDENT_CODE` https://cmake.org/cmake/help/latest/prop_tgt/POSITION_INDEPENDENT_CODE.html doesn't work? Do we need `-fexception` for `C_FLAGS`? Is `-fwasm-exceptions` https://emscripten.org/docs/porting/exceptions.html#webassembly-exception-handling-based-support better? Can we set this in `SetupCxxFlags.cmake` instead of here because `CMAKE_CXX_FLAGS` may be overwritten by users. ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and Review Comment: ```suggestion You need CMake and compilers etc. installed as per the normal build instructions. Before building with Emscripten, you also need to install Emscripten and ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten Review Comment: ```suggestion Cross compiling for WebAssembly with Emscripten ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same +versions of emsdk. + +.. code:: shell + + # install pyodide build tools. + # e.g. for version 0.24 of pyodide: + pip install pyodide-build==0.24 + +Then build with the ``ninja-release-emscripten-python`` cmake preset, +like below: + +.. code:: shell + + cmake --preset "ninja-release-emscripten-python" + ninja install + +This will install a built static library version of libarrow it into the +emscripten sysroot cache, meaning you can build things that depend on it +and they will find libarrow. Review Comment: ```suggestion and they will find ``libarrow``. ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same +versions of emsdk. + +.. code:: shell + + # install pyodide build tools. + # e.g. for version 0.24 of pyodide: + pip install pyodide-build==0.24 + +Then build with the ``ninja-release-emscripten-python`` cmake preset, +like below: + +.. code:: shell + + cmake --preset "ninja-release-emscripten-python" + ninja install + +This will install a built static library version of libarrow it into the +emscripten sysroot cache, meaning you can build things that depend on it +and they will find libarrow. + +e.g. if you want to build for pyodide, run the commands above, and then +go to ``arrow/python`` and run + +.. code:: shell + + pyodide build + +It should make a wheel targeting the currently enabled version of +pyodide (i.e. the version corresponding to the currently installed +``pyodide-build``) in the ``dist`` subdirectory. + +************** + Manual Build +************** + +If you want to manually build for emscripten, take a look at the +CMakePresets.json file in the arrow/cpp directory for a list of things +you will need to override. In particular you will need: + +#. Build dependencies set to ``BUNDLED``, so it uses properly cross + compiled build dependencies. + +#. ``CMAKE_TOOLCHAIN_FILE`` set to + ``arrow/cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake`` + +#. You will quite likely need to set ``ARROW_ENABLE_THREADING`` to ``OFF`` + for builds targeting single threaded emscripten environments such as Review Comment: ```suggestion for builds targeting single threaded Emscripten environments such as ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same +versions of emsdk. + +.. code:: shell + + # install pyodide build tools. + # e.g. for version 0.24 of pyodide: + pip install pyodide-build==0.24 + +Then build with the ``ninja-release-emscripten-python`` cmake preset, +like below: + +.. code:: shell + + cmake --preset "ninja-release-emscripten-python" + ninja install + +This will install a built static library version of libarrow it into the +emscripten sysroot cache, meaning you can build things that depend on it +and they will find libarrow. + +e.g. if you want to build for pyodide, run the commands above, and then +go to ``arrow/python`` and run + +.. code:: shell + + pyodide build + +It should make a wheel targeting the currently enabled version of +pyodide (i.e. the version corresponding to the currently installed +``pyodide-build``) in the ``dist`` subdirectory. + +************** + Manual Build +************** + +If you want to manually build for emscripten, take a look at the +CMakePresets.json file in the arrow/cpp directory for a list of things Review Comment: ```suggestion ``CMakePresets.json`` file in the ``arrow/cpp`` directory for a list of things ``` ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + +# # override default in emscripten which is to not use shared libs +set_property(GLOBAL PROPERTY TARGET_SUPPORTS_SHARED_LIBS TRUE) + +# if we leave the system name as Emscripten, then it reloads the original Emscripten.cmake every time a project() command +# is run, which does bad things like disabling shared libraries +set(CMAKE_SYSTEM_NAME EmscriptenOverrides) + +set(CMAKE_C_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") +set(CMAKE_CXX_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") + +#set(PYARROW_CPP_HOME "$ENV{ARROW_HOME}/lib") +#list(APPEND CMAKE_FIND_ROOT_PATH "${CMAKE_INSTALL_PREFIX}/cmake") + +set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE}) +set(Python3_LIBRARY $ENV{CPYTHONLIB}) +set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include) +set(Python3_EXECUTABLE) Review Comment: Why do we need set them? ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) Review Comment: `Emscripten.cmake` exists at `/usr/share/emscripten/cmake/Modules/Platform/Emscripten.cmake` on Debian. ( https://packages.debian.org/bookworm/all/emscripten/filelist ) So this doesn't work on Debian. It seems that Emscripten provides `emcmake`. Can we use it like `emcmake cmake ...` instead of specifying our toolchain file by `CMAKE_TOOLCHAIN_FILE` in `CMakePresets.json`? ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + +# # override default in emscripten which is to not use shared libs +set_property(GLOBAL PROPERTY TARGET_SUPPORTS_SHARED_LIBS TRUE) + +# if we leave the system name as Emscripten, then it reloads the original Emscripten.cmake every time a project() command +# is run, which does bad things like disabling shared libraries +set(CMAKE_SYSTEM_NAME EmscriptenOverrides) + +set(CMAKE_C_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") +set(CMAKE_CXX_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") + +#set(PYARROW_CPP_HOME "$ENV{ARROW_HOME}/lib") +#list(APPEND CMAKE_FIND_ROOT_PATH "${CMAKE_INSTALL_PREFIX}/cmake") Review Comment: Could you remove them? ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + +# # override default in emscripten which is to not use shared libs +set_property(GLOBAL PROPERTY TARGET_SUPPORTS_SHARED_LIBS TRUE) + +# if we leave the system name as Emscripten, then it reloads the original Emscripten.cmake every time a project() command +# is run, which does bad things like disabling shared libraries +set(CMAKE_SYSTEM_NAME EmscriptenOverrides) + +set(CMAKE_C_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") +set(CMAKE_CXX_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") + +#set(PYARROW_CPP_HOME "$ENV{ARROW_HOME}/lib") +#list(APPEND CMAKE_FIND_ROOT_PATH "${CMAKE_INSTALL_PREFIX}/cmake") + +set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE}) +set(Python3_LIBRARY $ENV{CPYTHONLIB}) +set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include) +set(Python3_EXECUTABLE) +set(CMAKE_SHARED_LIBRARY_CREATE_C_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_SHARED_LINKER_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") Review Comment: Do we need them? ########## cpp/cmake_modules/SetupCxxFlags.cmake: ########## @@ -24,7 +24,9 @@ include(CheckCXXSourceCompiles) message(STATUS "System processor: ${CMAKE_SYSTEM_PROCESSOR}") if(NOT DEFINED ARROW_CPU_FLAG) - if(CMAKE_SYSTEM_PROCESSOR MATCHES "AMD64|amd64|X86|x86|i[3456]86|x64") + if(CMAKE_SYSTEM_NAME MATCHES "Emscripten") Review Comment: Can we use `STREQUAL` instead of `MATCHES`? ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + +# # override default in emscripten which is to not use shared libs +set_property(GLOBAL PROPERTY TARGET_SUPPORTS_SHARED_LIBS TRUE) + +# if we leave the system name as Emscripten, then it reloads the original Emscripten.cmake every time a project() command +# is run, which does bad things like disabling shared libraries +set(CMAKE_SYSTEM_NAME EmscriptenOverrides) + +set(CMAKE_C_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") +set(CMAKE_CXX_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") + +#set(PYARROW_CPP_HOME "$ENV{ARROW_HOME}/lib") +#list(APPEND CMAKE_FIND_ROOT_PATH "${CMAKE_INSTALL_PREFIX}/cmake") + +set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE}) +set(Python3_LIBRARY $ENV{CPYTHONLIB}) +set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include) +set(Python3_EXECUTABLE) +set(CMAKE_SHARED_LIBRARY_CREATE_C_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_SHARED_LINKER_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_STRIP FALSE) Review Comment: Why do we need this? ########## cpp/cmake_modules/Emscripten/Platform/EmscriptenOverrides.cmake: ########## @@ -0,0 +1,56 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Force some variables for emscripten +# to disable things that won't work there + +# make us be on the platforms list for cmake +get_filename_component(PLATFORM_FOLDER_PARENT ${CMAKE_CURRENT_LIST_DIR} DIRECTORY) +list(APPEND CMAKE_MODULE_PATH "${PLATFORM_FOLDER_PARENT}") + +include($ENV{EMSDK}/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake) + +# ensure zlib is built with -fpic +# and force us to link to the version in emscripten ports +if(NOT EXISTS ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + execute_process(COMMAND embuilder --pic --force build zlib) +endif() +set(ZLIB_LIBRARY ${EMSCRIPTEN_SYSROOT}/lib/wasm32-emscripten/pic/libz.a) + +# # override default in emscripten which is to not use shared libs +set_property(GLOBAL PROPERTY TARGET_SUPPORTS_SHARED_LIBS TRUE) + +# if we leave the system name as Emscripten, then it reloads the original Emscripten.cmake every time a project() command +# is run, which does bad things like disabling shared libraries +set(CMAKE_SYSTEM_NAME EmscriptenOverrides) + +set(CMAKE_C_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") +set(CMAKE_CXX_FLAGS "-sUSE_ZLIB=1 -sSIDE_MODULE=1 -fPIC -fexceptions") + +#set(PYARROW_CPP_HOME "$ENV{ARROW_HOME}/lib") +#list(APPEND CMAKE_FIND_ROOT_PATH "${CMAKE_INSTALL_PREFIX}/cmake") + +set(Python3_INCLUDE_DIR $ENV{PYTHONINCLUDE}) +set(Python3_LIBRARY $ENV{CPYTHONLIB}) +set(Python3_NumPy_INCLUDE_DIR $ENV{NUMPY_LIB}/core/include) +set(Python3_EXECUTABLE) +set(CMAKE_SHARED_LIBRARY_CREATE_C_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_SHARED_LINKER_FLAGS "-sUSE_ZLIB=1 -sWASM_BIGINT=1 -fexceptions") +set(CMAKE_STRIP FALSE) + +set(ENV{_PYTHON_SYSCONFIGDATA_NAME} $ENV{SYSCONFIG_NAME}) Review Comment: Can we remove this? ########## cpp/cmake_modules/ThirdpartyToolchain.cmake: ########## @@ -1614,6 +1619,9 @@ macro(build_thrift) if(DEFINED BOOST_ROOT) list(APPEND THRIFT_CMAKE_ARGS "-DBOOST_ROOT=${BOOST_ROOT}") endif() + if(DEFINED Boost_INCLUDE_DIR) + list(APPEND THRIFT_CMAKE_ARGS "-DBoost_INCLUDE_DIR=${Boost_INCLUDE_DIR}") Review Comment: How about using `BOOST_INCLUDE_DIR` instead of `Boost_INCLUDE_DIR` because `BOOST_...` is a hint variable https://cmake.org/cmake/help/latest/module/FindBoost.html#hints and `Boost_...` is a cache result variable https://cmake.org/cmake/help/latest/module/FindBoost.html#cache-variables ? ```suggestion list(APPEND THRIFT_CMAKE_ARGS "-DBOOST_INCLUDE_DIR=${Boost_INCLUDE_DIR}") ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you Review Comment: ```suggestion If you want to build PyArrow for `Pyodide <https://pyodide.org>`_, you ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same +versions of emsdk. + +.. code:: shell + + # install pyodide build tools. + # e.g. for version 0.24 of pyodide: + pip install pyodide-build==0.24 + +Then build with the ``ninja-release-emscripten-python`` cmake preset, +like below: + +.. code:: shell + + cmake --preset "ninja-release-emscripten-python" + ninja install + +This will install a built static library version of libarrow it into the +emscripten sysroot cache, meaning you can build things that depend on it Review Comment: ```suggestion Emscripten sysroot cache, meaning you can build things that depend on it ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same Review Comment: ```suggestion same version of Python that Pyodide is built for, along with the same ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same +versions of emsdk. + +.. code:: shell + + # install pyodide build tools. + # e.g. for version 0.24 of pyodide: + pip install pyodide-build==0.24 + +Then build with the ``ninja-release-emscripten-python`` cmake preset, +like below: + +.. code:: shell + + cmake --preset "ninja-release-emscripten-python" + ninja install + +This will install a built static library version of libarrow it into the +emscripten sysroot cache, meaning you can build things that depend on it +and they will find libarrow. + +e.g. if you want to build for pyodide, run the commands above, and then Review Comment: ```suggestion e.g. if you want to build for Pyodide, run the commands above, and then ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same +versions of emsdk. + +.. code:: shell + + # install pyodide build tools. + # e.g. for version 0.24 of pyodide: + pip install pyodide-build==0.24 + +Then build with the ``ninja-release-emscripten-python`` cmake preset, +like below: + +.. code:: shell + + cmake --preset "ninja-release-emscripten-python" + ninja install + +This will install a built static library version of libarrow it into the Review Comment: ```suggestion This will install a built static library version of ``libarrow`` it into the ``` ########## docs/source/developers/cpp/emscripten.rst: ########## @@ -0,0 +1,100 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + + +.. highlight:: console .. _developers-cpp-emscripten: + +################################################# + Cross compiling for Webassembly with Emscripten +################################################# + +*************** + Prerequisites +*************** +You need cmake and compilers etc. installed as per the normal build instructions. Before building with emscripten, you also need to install emscripten and +activate it using the commands below (see https://emscripten.org/docs/getting_started/downloads.html for details). + +.. code:: shell + + git clone https://github.com/emscripten-core/emsdk.git + cd emsdk + # replace <version> with the desired EMSDK version. + # e.g. for pyodide 0.24, you need EMSDK version 3.1.45 + ./emsdk install <version> + ./emsdk activate <version> + source ./emsdk_env.sh + +If you want to build pyarrow for `pyodide <https://pyodide.org>`_, you +need ``pyodide-build`` installed via ``pip``, and to be running with the +same version of python that pyodide is built for, along with the same +versions of emsdk. + +.. code:: shell + + # install pyodide build tools. + # e.g. for version 0.24 of pyodide: + pip install pyodide-build==0.24 + +Then build with the ``ninja-release-emscripten-python`` cmake preset, +like below: + +.. code:: shell + + cmake --preset "ninja-release-emscripten-python" + ninja install + +This will install a built static library version of libarrow it into the +emscripten sysroot cache, meaning you can build things that depend on it +and they will find libarrow. + +e.g. if you want to build for pyodide, run the commands above, and then +go to ``arrow/python`` and run + +.. code:: shell + + pyodide build + +It should make a wheel targeting the currently enabled version of +pyodide (i.e. the version corresponding to the currently installed +``pyodide-build``) in the ``dist`` subdirectory. + +************** + Manual Build +************** + +If you want to manually build for emscripten, take a look at the Review Comment: ```suggestion If you want to manually build for Emscripten, take a look at the ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
