This is an automated email from the ASF dual-hosted git repository.
chaokunyang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fory.git
The following commit(s) were added to refs/heads/main by this push:
new 61e986095 feat(python): implement cython bazel build directly without
3rd deps (#2936)
61e986095 is described below
commit 61e9860958dc7e6f1f927ad7d8adeec0505a5e5c
Author: Shawn Yang <[email protected]>
AuthorDate: Thu Nov 27 14:00:22 2025 +0800
feat(python): implement cython bazel build directly without 3rd deps (#2936)
<!--
**Thanks for contributing to Apache Fory™.**
**If this is your first time opening a PR on fory, you can refer to
[CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).**
Contribution Checklist
- The **Apache Fory™** community has requirements on the naming of pr
titles. You can also find instructions in
[CONTRIBUTING.md](https://github.com/apache/fory/blob/main/CONTRIBUTING.md).
- Apache Fory™ has a strong focus on performance. If the PR you submit
will have an impact on performance, please benchmark it first and
provide the benchmark result here.
-->
## Why?
implement cython bazel build directly without using grpc cython rule
## Related issues
#2279
#2285
## Does this PR introduce any user-facing change?
<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fory/issues/new/choose) describing the
need to do so and update the document if necessary.
Delete section if not applicable.
-->
- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?
## Benchmark
<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.
Delete section if not applicable.
-->
---
AGENTS.md | 25 +-
BUILD | 2 +-
LICENSE | 10 +-
WORKSPACE | 4 +-
bazel/cython.BUILD | 28 +++
bazel/cython_library.bzl | 75 ++++++
bazel/fory_deps_setup.bzl | 13 +-
bazel/grpc-cython-copts.patch | 37 ---
bazel/grpc-python.patch | 12 -
WORKSPACE => bazel/py/BUILD | 15 +-
bazel/py/BUILD.tpl | 30 +++
bazel/py/python_configure.bzl | 384 +++++++++++++++++++++++++++++
bazel/py/variety.tpl | 28 +++
cpp/fory/python/BUILD | 2 +-
LICENSE => licenses/LICENSE-tensorflow.txt | 154 ++++--------
15 files changed, 624 insertions(+), 195 deletions(-)
diff --git a/AGENTS.md b/AGENTS.md
index a8d02b45f..44ac62da1 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -57,20 +57,20 @@ mvn -T16 test -Dtest=org.apache.fory.TestClass#testMethod
- private methods should be put last in class def, before private fields.
```bash
-# Prepare for build
-pip install pyarrow
-
# Build C++ library
-bazel build //...
+bazel build //cpp/...
+
+# Build Cython extensions
+bazel build //:cp_fory_so
# Run tests
-bazel test $(bazel query //...)
+bazel test $(bazel query //cpp/...)
# Run serialization tests
bazel test $(bazel query //cpp/fory/serialization/...)
# Run specific test
-bazel test //fory/util:buffer_test
+bazel test //cpp/fory/util:buffer_test
# format c++ code
clang-format -i $file
@@ -383,17 +383,20 @@ Fory serialization for every language is implemented
independently to minimize t
#### Bazel
-`bazel` dir provide build support for fory c++ and cython:
+`bazel` dir provides build support for fory C++ and Cython:
-- `bazel/arrow`: build rules to get arrow shared libraries based on bazel
template
-- `grpc-cython-copts.patch/grpc-python.patch`: patch for grpc to add
`pyx_library` for cython.
+- `bazel/fory_deps_setup.bzl`: setup external dependencies for bazel build
+- `bazel/cython_library.bzl`: `pyx_library` rule for building Cython extensions
+- `bazel/cython.BUILD`: build file for external Cython dependency
+- `bazel/py/python_configure.bzl`: repository rule for Python autoconfiguration
+- `bazel/py/BUILD.tpl`: template for generated Python configuration BUILD file
+- `bazel/py/variety.tpl`: template for Python version-specific configuration
#### C++
- `cpp/fory/row`: Row format data structures
- `cpp/fory/meta`: Compile-time reflection utilities for extract struct fields
information.
- `cpp/fory/encoder`: Row format encoder and decoder
-- `cpp/fory/columnar`: Interoperation between fory row format and apache arrow
columnar format
- `cpp/fory/util`: Common utilities
- `cpp/fory/util/buffer.h`: Buffer for reading and writing data
- `cpp/fory/util/bit_util.h`: utilities for bit manipulation
@@ -560,5 +563,5 @@ refactor(java): unify serialization exceptions hierarchy
perf(cpp): optimize buffer allocation in encoder
test(integration): add cross-language reference cycle tests
ci: update build matrix for latest JDK versions
-chore(deps): update arrow dependency to 15.0.0
+chore(deps): update guava dependency to 32.0.0
```
diff --git a/BUILD b/BUILD
index 34d24480c..286cf58f6 100644
--- a/BUILD
+++ b/BUILD
@@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.
-load("@com_github_grpc_grpc//bazel:cython_library.bzl", "pyx_library")
+load("//bazel:cython_library.bzl", "pyx_library")
load("@compile_commands_extractor//:refresh_compile_commands.bzl",
"refresh_compile_commands")
diff --git a/LICENSE b/LICENSE
index 0a90a1b47..d39c69473 100644
--- a/LICENSE
+++ b/LICENSE
@@ -243,12 +243,14 @@ The text of each license is the standard Apache 2.0
license.
Files:
ci/format.sh
bazel/fory_deps_setup.bzl
- bazel/grpc-cython-copts.patch
- bazel/grpc-python.patch
-* tensorflow (https://github.com/tensorflow/tensorflow)
+* grpc (https://github.com/grpc/grpc) and tensorflow
(https://github.com/tensorflow/tensorflow)
Files:
- bazel/arrow/pyarrow_configure.bzl
+ bazel/cython_library.bzl
+ bazel/cython.BUILD
+ bazel/py/python_configure.bzl
+ bazel/py/BUILD.tpl
+ bazel/py/variety.tpl
================================================================
diff --git a/WORKSPACE b/WORKSPACE
index ce27cf782..9e268ac25 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -21,8 +21,6 @@ load("//bazel:fory_deps_setup.bzl", "setup_deps")
setup_deps()
load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
-load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
-load("@com_github_grpc_grpc//third_party/py:python_configure.bzl",
"python_configure")
+load("//bazel/py:python_configure.bzl", "python_configure")
bazel_skylib_workspace()
python_configure(name="local_config_python")
-grpc_deps()
diff --git a/bazel/cython.BUILD b/bazel/cython.BUILD
new file mode 100644
index 000000000..1c4d33997
--- /dev/null
+++ b/bazel/cython.BUILD
@@ -0,0 +1,28 @@
+# Adapted from tensorflow/third_party/cython.BUILD
+
+py_library(
+ name="cython_lib",
+ srcs=glob(
+ ["Cython/**/*.py"],
+ exclude=[
+ "**/Tests/*.py",
+ ],
+ ) + ["cython.py"],
+ data=glob([
+ "Cython/**/*.pyx",
+ "Cython/Utility/*.*",
+ "Cython/Includes/**/*.pxd",
+ ]),
+ srcs_version="PY2AND3",
+ visibility=["//visibility:public"],
+)
+
+# May not be named "cython", since that conflicts with Cython/ on OSX
+py_binary(
+ name="cython_binary",
+ srcs=["cython.py"],
+ main="cython.py",
+ srcs_version="PY2AND3",
+ visibility=["//visibility:public"],
+ deps=["cython_lib"],
+)
diff --git a/bazel/cython_library.bzl b/bazel/cython_library.bzl
new file mode 100644
index 000000000..11ed111f2
--- /dev/null
+++ b/bazel/cython_library.bzl
@@ -0,0 +1,75 @@
+# Adapted from tensorflow/tensorflow/core/platform/default/build_config.bzl
+# and grpc/bazel/cython_library.bzl
+"""Custom rules for building Cython extensions"""
+
+def pyx_library(name, deps = [], cc_kwargs = {}, py_deps = [], srcs = [],
**kwargs):
+ """Compiles a group of .pyx / .pxd / .py files.
+
+ First runs Cython to create .cpp files for each input .pyx or .py + .pxd
+ pair. Then builds a shared object for each, passing "deps" and
`**cc_kwargs`
+ to each cc_binary rule (includes Python headers by default). Finally,
creates
+ a py_library rule with the shared objects and any pure Python "srcs", with
py_deps
+ as its dependencies; the shared objects can be imported like normal Python
files.
+
+ Args:
+ name: Name for the rule.
+ deps: C/C++ dependencies of the Cython (e.g. Numpy headers).
+ cc_kwargs: cc_binary extra arguments such as copts, linkstatic,
linkopts, features
+ py_deps: Pure Python dependencies of the final library.
+ srcs: .py, .pyx, or .pxd files to either compile or pass through.
+ **kwargs: Extra keyword arguments passed to the py_library.
+ """
+
+ # First filter out files that should be run compiled vs. passed through.
+ py_srcs = []
+ pyx_srcs = []
+ pxd_srcs = []
+ for src in srcs:
+ if src.endswith(".pyx") or (src.endswith(".py") and
+ src[:-3] + ".pxd" in srcs):
+ pyx_srcs.append(src)
+ elif src.endswith(".py"):
+ py_srcs.append(src)
+ else:
+ pxd_srcs.append(src)
+ if src.endswith("__init__.py"):
+ pxd_srcs.append(src)
+
+ # Invoke cython to produce the shared object libraries.
+ for filename in pyx_srcs:
+ native.genrule(
+ name = filename + "_cython_translation",
+ srcs = [filename],
+ outs = [filename.split(".")[0] + ".cpp"],
+ # Optionally use PYTHON_BIN_PATH on Linux platforms so that python
3
+ # works. Windows has issues with cython_binary so skip
PYTHON_BIN_PATH.
+ cmd =
+ "PYTHONHASHSEED=0 $(location @cython//:cython_binary) --cplus
$(SRCS) --output-file $(OUTS)",
+ tools = ["@cython//:cython_binary"] + pxd_srcs,
+ )
+
+ shared_objects = []
+ for src in pyx_srcs:
+ stem = src.split(".")[0]
+ shared_object_name = stem + ".so"
+ native.cc_binary(
+ name = cc_kwargs.pop("name", shared_object_name),
+ srcs = [stem + ".cpp"] + cc_kwargs.pop("srcs", []),
+ deps = deps + ["@local_config_python//:python_headers"] +
cc_kwargs.pop("deps", []),
+ linkshared = cc_kwargs.pop("linkshared", 1),
+ **cc_kwargs
+ )
+ shared_objects.append(shared_object_name)
+
+ data = shared_objects[:]
+ data += kwargs.pop("data", [])
+
+ # Now create a py_library with these shared objects as data.
+ native.py_library(
+ name = name,
+ srcs = py_srcs,
+ deps = py_deps,
+ srcs_version = "PY2AND3",
+ data = data,
+ **kwargs
+ )
diff --git a/bazel/fory_deps_setup.bzl b/bazel/fory_deps_setup.bzl
index a8131bd3c..95e282a7d 100644
--- a/bazel/fory_deps_setup.bzl
+++ b/bazel/fory_deps_setup.bzl
@@ -130,21 +130,12 @@ def setup_deps():
strip_prefix = "rules_python-0.8.0",
url =
"https://github.com/bazelbuild/rules_python/archive/refs/tags/0.8.0.tar.gz";,
)
- auto_http_archive(
- name = "com_github_grpc_grpc",
- # NOTE: If you update this, also update @boringssl's hash.
- url = "https://github.com/grpc/grpc/archive/refs/tags/v1.46.6.tar.gz";,
- sha256 =
"6514b3e6eab9e9c7017304512d4420387a47b1a9c5caa986643692977ed44e8a",
- patches = [
- "//bazel:grpc-cython-copts.patch",
- "//bazel:grpc-python.patch",
- ],
- )
+ # Cython for building Python extensions
# leary cython failed with: `found 'CKeyValueMetadata'`
# see https://github.com/apache/arrow/issues/28629
auto_http_archive(
name = "cython",
- build_file = "@com_github_grpc_grpc//third_party:cython.BUILD",
+ build_file = "@fory//bazel:cython.BUILD",
url =
"https://github.com/cython/cython/releases/download/3.1.1/cython-3.1.1.tar.gz";,
sha256 =
"505ccd413669d5132a53834d792c707974248088c4f60c497deb1b416e366397",
)
diff --git a/bazel/grpc-cython-copts.patch b/bazel/grpc-cython-copts.patch
deleted file mode 100644
index 0d75f037d..000000000
--- a/bazel/grpc-cython-copts.patch
+++ /dev/null
@@ -1,37 +0,0 @@
-diff --git bazel/cython_library.bzl bazel/cython_library.bzl
---- bazel/cython_library.bzl
-+++ bazel/cython_library.bzl
-@@ -10,15 +10,16 @@
--def pyx_library(name, deps = [], py_deps = [], srcs = [], **kwargs):
-+def pyx_library(name, deps = [], cc_kwargs = {}, py_deps = [], srcs = [],
**kwargs):
- """Compiles a group of .pyx / .pxd / .py files.
-
- First runs Cython to create .cpp files for each input .pyx or .py + .pxd
-- pair. Then builds a shared object for each, passing "deps" to each
cc_binary
-- rule (includes Python headers by default). Finally, creates a py_library
rule
-- with the shared objects and any pure Python "srcs", with py_deps as its
-- dependencies; the shared objects can be imported like normal Python files.
-+ pair. Then builds a shared object for each, passing "deps" and
`**cc_kwargs`
-+ to each cc_binary rule (includes Python headers by default). Finally,
creates
-+ a py_library rule with the shared objects and any pure Python "srcs",
with py_deps
-+ as its dependencies; the shared objects can be imported like normal
Python files.
-
- Args:
- name: Name for the rule.
- deps: C/C++ dependencies of the Cython (e.g. Numpy headers).
-+ cc_kwargs: cc_binary extra arguments such as copts, linkstatic,
linkopts, features
-@@ -57,7 +59,8 @@ def pyx_library(name, deps = [], py_deps = [], srcs = [],
**kwargs):
-- shared_object_name = stem + ".so"
-+ shared_object_name = stem + ".so"
- native.cc_binary(
-- name = shared_object_name,
-+ name = cc_kwargs.pop("name", shared_object_name),
-- srcs = [stem + ".cpp"],
-+ srcs = [stem + ".cpp"] + cc_kwargs.pop("srcs", []),
-- deps = deps + ["@local_config_python//:python_headers"],
-+ deps = deps + ["@local_config_python//:python_headers"] +
cc_kwargs.pop("deps", []),
-- linkshared = 1,
-+ linkshared = cc_kwargs.pop("linkshared", 1),
-+ **cc_kwargs
- )
---
\ No newline at end of file
diff --git a/bazel/grpc-python.patch b/bazel/grpc-python.patch
deleted file mode 100644
index 8db258d1e..000000000
--- a/bazel/grpc-python.patch
+++ /dev/null
@@ -1,12 +0,0 @@
-diff --git third_party/py/python_configure.bzl
third_party/py/python_configure.bzl
---- third_party/py/python_configure.bzl
-+++ third_party/py/python_configure.bzl
-@@ -345,7 +345,7 @@ def _python_autoconf_impl(repository_ctx):
- repository_ctx,
- "_python3",
- _PYTHON3_BIN_PATH,
-- "python3",
-+ "python3" if not _is_windows(repository_ctx) else "python.exe",
- _PYTHON3_LIB_PATH,
- False
- )
\ No newline at end of file
diff --git a/WORKSPACE b/bazel/py/BUILD
similarity index 67%
copy from WORKSPACE
copy to bazel/py/BUILD
index ce27cf782..1fc80965e 100644
--- a/WORKSPACE
+++ b/bazel/py/BUILD
@@ -15,14 +15,9 @@
# specific language governing permissions and limitations
# under the License.
-workspace(name = "fory")
+# Package for Python configuration templates
-load("//bazel:fory_deps_setup.bzl", "setup_deps")
-setup_deps()
-
-load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
-load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
-load("@com_github_grpc_grpc//third_party/py:python_configure.bzl",
"python_configure")
-bazel_skylib_workspace()
-python_configure(name="local_config_python")
-grpc_deps()
+exports_files([
+ "BUILD.tpl",
+ "variety.tpl",
+])
diff --git a/bazel/py/BUILD.tpl b/bazel/py/BUILD.tpl
new file mode 100644
index 000000000..0677a083b
--- /dev/null
+++ b/bazel/py/BUILD.tpl
@@ -0,0 +1,30 @@
+# Adapted from tensorflow/third_party/py/ and grpc/third_party/py/
+
+package(default_visibility=["//visibility:public"])
+
+config_setting(
+ name="windows",
+ values={"cpu": "x64_windows"},
+ visibility=["//visibility:public"],
+)
+
+config_setting(
+ name="python3",
+ flag_values = {"@rules_python//python:python_version": "PY3"}
+)
+
+cc_library(
+ name = "python_lib",
+ deps = select({
+ ":python3": ["//_python3:_python3_lib"],
+ "//conditions:default": ["not-existing.lib"],
+ })
+)
+
+cc_library(
+ name = "python_headers",
+ deps = select({
+ ":python3": ["//_python3:_python3_headers"],
+ "//conditions:default": ["not-existing.headers"],
+ })
+)
diff --git a/bazel/py/python_configure.bzl b/bazel/py/python_configure.bzl
new file mode 100644
index 000000000..8181b712e
--- /dev/null
+++ b/bazel/py/python_configure.bzl
@@ -0,0 +1,384 @@
+# Adapted from tensorflow/third_party/py/ and grpc/third_party/py/
+"""Repository rule for Python autoconfiguration.
+
+`python_configure` depends on the following environment variables:
+
+ * `PYTHON3_BIN_PATH`: location of python binary.
+ * `PYTHON3_LIB_PATH`: Location of python libraries.
+"""
+
+_BAZEL_SH = "BAZEL_SH"
+_PYTHON3_BIN_PATH = "PYTHON3_BIN_PATH"
+_PYTHON3_LIB_PATH = "PYTHON3_LIB_PATH"
+
+_HEADERS_HELP = (
+ "Are Python headers installed? Try installing " +
+ "python3-dev on Debian-based systems. Try python3-devel " +
+ "on Redhat-based systems."
+)
+
+def _tpl(repository_ctx, tpl, substitutions = {}, out = None):
+ if not out:
+ out = tpl
+ repository_ctx.template(
+ out,
+ Label("//bazel/py:%s.tpl" % tpl),
+ substitutions,
+ )
+
+def _fail(msg):
+ """Output failure message when auto configuration fails."""
+ red = "\033[0;31m"
+ no_color = "\033[0m"
+ fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
+
+def _is_windows(repository_ctx):
+ """Returns true if the host operating system is windows."""
+ os_name = repository_ctx.os.name.lower()
+ return os_name.find("windows") != -1
+
+def _execute(
+ repository_ctx,
+ cmdline,
+ error_msg = None,
+ error_details = None,
+ empty_stdout_fine = False):
+ """Executes an arbitrary shell command.
+
+ Args:
+ repository_ctx: the repository_ctx object
+ cmdline: list of strings, the command to execute
+ error_msg: string, a summary of the error if the command fails
+ error_details: string, details about the error or steps to fix it
+ empty_stdout_fine: bool, if True, an empty stdout result is fine,
otherwise
+ it's an error
+ Return:
+ the result of repository_ctx.execute(cmdline)
+ """
+ result = repository_ctx.execute(cmdline)
+ if result.stderr or not (empty_stdout_fine or result.stdout):
+ _fail("\n".join([
+ error_msg.strip() if error_msg else "Repository command failed",
+ result.stderr.strip(),
+ error_details if error_details else "",
+ ]))
+ else:
+ return result
+
+def _read_dir(repository_ctx, src_dir):
+ """Returns a string with all files in a directory.
+
+ Finds all files inside a directory, traversing subfolders and following
+ symlinks. The returned string contains the full path of all files
+ separated by line breaks.
+ """
+ if _is_windows(repository_ctx):
+ src_dir = src_dir.replace("/", "\\")
+ find_result = _execute(
+ repository_ctx,
+ ["cmd.exe", "/c", "dir", src_dir, "/b", "/s", "/a-d"],
+ empty_stdout_fine = True,
+ )
+
+ # src_files will be used in genrule.outs where the paths must
+ # use forward slashes.
+ return find_result.stdout.replace("\\", "/")
+ else:
+ find_result = _execute(
+ repository_ctx,
+ ["find", src_dir, "-follow", "-type", "f"],
+ empty_stdout_fine = True,
+ )
+ return find_result.stdout
+
+def _genrule(src_dir, genrule_name, command, outs):
+ """Returns a string with a genrule.
+
+ Genrule executes the given command and produces the given outputs.
+ """
+ return ("genrule(\n" + ' name = "' + genrule_name + '",\n' +
+ " outs = [\n" + outs + "\n ],\n" + ' cmd = """\n' +
+ command + '\n """,\n' + ")\n")
+
+def _normalize_path(path):
+ """Returns a path with '/' and remove the trailing slash."""
+ path = path.replace("\\", "/")
+ if path[-1] == "/":
+ path = path[:-1]
+ return path
+
+def _symlink_genrule_for_dir(
+ repository_ctx,
+ src_dir,
+ dest_dir,
+ genrule_name,
+ src_files = [],
+ dest_files = []):
+ """Returns a genrule to symlink(or copy if on Windows) a set of files.
+
+ If src_dir is passed, files will be read from the given directory;
otherwise
+ we assume files are in src_files and dest_files
+ """
+ if src_dir != None:
+ src_dir = _normalize_path(src_dir)
+ dest_dir = _normalize_path(dest_dir)
+ files = "\n".join(
+ sorted(_read_dir(repository_ctx, src_dir).splitlines()),
+ )
+
+ # Create a list with the src_dir stripped to use for outputs.
+ dest_files = files.replace(src_dir, "").splitlines()
+ src_files = files.splitlines()
+ command = []
+ outs = []
+ for i in range(len(dest_files)):
+ if dest_files[i] != "":
+ # If we have only one file to link we do not want to use the
dest_dir, as
+ # $(@D) will include the full path to the file.
+ dest = "$(@D)/" + dest_dir + dest_files[i] if len(
+ dest_files,
+ ) != 1 else "$(@D)/" + dest_files[i]
+
+ # On Windows, symlink is not supported, so we just copy all the
files.
+ cmd = "cp -f" if _is_windows(repository_ctx) else "ln -s"
+ command.append(cmd + ' "%s" "%s"' % (src_files[i], dest))
+ outs.append(' "' + dest_dir + dest_files[i] + '",')
+ return _genrule(
+ src_dir,
+ genrule_name,
+ " && ".join(command),
+ "\n".join(outs),
+ )
+
+def _get_python_bin(repository_ctx, bin_path_key, default_bin_path,
allow_absent):
+ """Gets the python bin path."""
+ python_bin = repository_ctx.os.environ.get(bin_path_key, default_bin_path)
+ if not repository_ctx.path(python_bin).exists:
+ # It's a command, use 'which' to find its path.
+ python_bin_path = repository_ctx.which(python_bin)
+ else:
+ # It's a path, use it as it is.
+ python_bin_path = python_bin
+ if python_bin_path != None:
+ return str(python_bin_path)
+ if not allow_absent:
+ _fail("Cannot find python in PATH, please make sure " +
+ "python is installed and add its directory in PATH, or --define
" +
+ "%s='/something/else'.\nPATH=%s" %
+ (bin_path_key, repository_ctx.os.environ.get("PATH", "")))
+ else:
+ return None
+
+def _get_bash_bin(repository_ctx):
+ """Gets the bash bin path."""
+ bash_bin = repository_ctx.os.environ.get(_BAZEL_SH)
+ if bash_bin != None:
+ return bash_bin
+ else:
+ bash_bin_path = repository_ctx.which("bash")
+ if bash_bin_path != None:
+ return str(bash_bin_path)
+ else:
+ _fail(
+ "Cannot find bash in PATH, please make sure " +
+ "bash is installed and add its directory in PATH, or --define
" +
+ "%s='/path/to/bash'.\nPATH=%s" %
+ (_BAZEL_SH, repository_ctx.os.environ.get("PATH", "")),
+ )
+
+def _get_python_lib(repository_ctx, python_bin, lib_path_key):
+ """Gets the python lib path."""
+ python_lib = repository_ctx.os.environ.get(lib_path_key)
+ if python_lib != None:
+ return python_lib
+ print_lib = (
+ "<<END\n" + "from __future__ import print_function\n" +
+ "import site\n" + "import os\n" + "\n" + "try:\n" +
+ " input = raw_input\n" + "except NameError:\n" + " pass\n" + "\n" +
+ "python_paths = []\n" + "if os.getenv('PYTHONPATH') is not None:\n" +
+ " python_paths = os.getenv('PYTHONPATH').split(':')\n" + "try:\n" +
+ " library_paths = site.getsitepackages()\n" +
+ "except AttributeError:\n" +
+ " import sysconfig\n" +
+ " library_paths = [sysconfig.get_path('purelib')]\n" +
+ "all_paths = set(python_paths + library_paths)\n" + "paths = []\n" +
+ "for path in all_paths:\n" + " if os.path.isdir(path):\n" +
+ " paths.append(path)\n" + "if len(paths) >=1:\n" +
+ " print(paths[0])\n" + "END"
+ )
+ cmd = '"%s" - %s' % (python_bin, print_lib)
+ result = repository_ctx.execute([_get_bash_bin(repository_ctx), "-c", cmd])
+ return result.stdout.strip("\n")
+
+def _check_python_lib(repository_ctx, python_lib):
+ """Checks the python lib path."""
+ cmd = 'test -d "%s" -a -x "%s"' % (python_lib, python_lib)
+ result = repository_ctx.execute([_get_bash_bin(repository_ctx), "-c", cmd])
+ if result.return_code == 1:
+ _fail("Invalid python library path: %s" % python_lib)
+
+def _check_python_bin(repository_ctx, python_bin, bin_path_key, allow_absent):
+ """Checks the python bin path."""
+ cmd = '[[ -x "%s" ]] && [[ ! -d "%s" ]]' % (python_bin, python_bin)
+ result = repository_ctx.execute([_get_bash_bin(repository_ctx), "-c", cmd])
+ if result.return_code == 1:
+ if not allow_absent:
+ _fail("--define %s='%s' is not executable. Is it the python
binary?" %
+ (bin_path_key, python_bin))
+ else:
+ return None
+ return True
+
+def _get_python_include(repository_ctx, python_bin):
+ """Gets the python include path."""
+ result = _execute(
+ repository_ctx,
+ [
+ python_bin,
+ "-c",
+ "from __future__ import print_function;" +
+ "import sysconfig;" +
+ "print(sysconfig.get_path('include'))",
+ ],
+ error_msg = "Problem getting python include path for
{}.".format(python_bin),
+ error_details = (
+ "Is the Python binary path set up right? " + "(See ./configure or
" +
+ python_bin + ".) " + _HEADERS_HELP
+ ),
+ )
+ include_path = result.stdout.splitlines()[0]
+ _execute(
+ repository_ctx,
+ [
+ python_bin,
+ "-c",
+ "import os;" +
+ "main_header = os.path.join(r'{}',
'Python.h');".format(include_path) +
+ "assert os.path.exists(main_header), main_header + ' does not
exist.'",
+ ],
+ error_msg = "Unable to find Python headers for {}".format(python_bin),
+ error_details = _HEADERS_HELP,
+ empty_stdout_fine = True,
+ )
+ return include_path
+
+def _get_python_import_lib_name(repository_ctx, python_bin, bin_path_key):
+ """Get Python import library name (pythonXY.lib) on Windows."""
+ result = _execute(
+ repository_ctx,
+ [
+ python_bin,
+ "-c",
+ "import sys;" + 'print("python" + str(sys.version_info[0]) + ' +
+ ' str(sys.version_info[1]) + ".lib")',
+ ],
+ error_msg = "Problem getting python import library.",
+ error_details = ("Is the Python binary path set up right? " +
+ "(See ./configure or " + bin_path_key + ".) "),
+ )
+ return result.stdout.splitlines()[0]
+
+def _create_single_version_package(
+ repository_ctx,
+ variety_name,
+ bin_path_key,
+ default_bin_path,
+ lib_path_key,
+ allow_absent):
+ """Creates the repository containing files set up to build with Python."""
+ empty_include_rule = "filegroup(\n name=\"{}_include\",\n
srcs=[],\n)".format(variety_name)
+
+ python_bin = _get_python_bin(repository_ctx, bin_path_key,
default_bin_path, allow_absent)
+ if (python_bin == None or
+ _check_python_bin(repository_ctx,
+ python_bin,
+ bin_path_key,
+ allow_absent) == None) and allow_absent:
+ python_include_rule = empty_include_rule
+ else:
+ python_lib = _get_python_lib(repository_ctx, python_bin, lib_path_key)
+ _check_python_lib(repository_ctx, python_lib)
+ python_include = _get_python_include(repository_ctx, python_bin)
+ python_include_rule = _symlink_genrule_for_dir(
+ repository_ctx,
+ python_include,
+ "{}_include".format(variety_name),
+ "{}_include".format(variety_name),
+ )
+ python_import_lib_genrule = ""
+
+ # To build Python C/C++ extension on Windows, we need to link to python
import library pythonXY.lib
+ # See https://docs.python.org/3/extending/windows.html
+ if _is_windows(repository_ctx):
+ python_include = _normalize_path(python_include)
+ python_import_lib_name = _get_python_import_lib_name(
+ repository_ctx,
+ python_bin,
+ bin_path_key,
+ )
+ python_import_lib_src = python_include.rsplit(
+ "/",
+ 1,
+ )[0] + "/libs/" + python_import_lib_name
+ python_import_lib_genrule = _symlink_genrule_for_dir(
+ repository_ctx,
+ None,
+ "",
+ "{}_import_lib".format(variety_name),
+ [python_import_lib_src],
+ [python_import_lib_name],
+ )
+ _tpl(
+ repository_ctx,
+ "variety",
+ {
+ "%{PYTHON_INCLUDE_GENRULE}": python_include_rule,
+ "%{PYTHON_IMPORT_LIB_GENRULE}": python_import_lib_genrule,
+ "%{VARIETY_NAME}": variety_name,
+ },
+ out = "{}/BUILD".format(variety_name),
+ )
+
+def _python_autoconf_impl(repository_ctx):
+ """Implementation of the python_autoconf repository rule."""
+ _create_single_version_package(
+ repository_ctx,
+ "_python3",
+ _PYTHON3_BIN_PATH,
+ "python3" if not _is_windows(repository_ctx) else "python.exe",
+ _PYTHON3_LIB_PATH,
+ False
+ )
+ _tpl(repository_ctx, "BUILD")
+
+python_configure = repository_rule(
+ implementation = _python_autoconf_impl,
+ environ = [
+ _BAZEL_SH,
+ _PYTHON3_BIN_PATH,
+ _PYTHON3_LIB_PATH,
+ ],
+ attrs = {
+ "_build_tpl": attr.label(
+ default = Label("//bazel/py:BUILD.tpl"),
+ allow_single_file = True,
+ ),
+ "_variety_tpl": attr.label(
+ default = Label("//bazel/py:variety.tpl"),
+ allow_single_file = True,
+ ),
+ },
+)
+"""Detects and configures the local Python.
+
+It expects the system have a working Python 3 installation.
+
+Add the following to your WORKSPACE FILE:
+
+```python
+python_configure(name = "local_config_python")
+```
+
+Args:
+ name: A unique name for this workspace rule.
+"""
diff --git a/bazel/py/variety.tpl b/bazel/py/variety.tpl
new file mode 100644
index 000000000..2e1725845
--- /dev/null
+++ b/bazel/py/variety.tpl
@@ -0,0 +1,28 @@
+# Adapted from tensorflow/third_party/py/ and grpc/third_party/py/
+
+package(default_visibility=["//visibility:public"])
+
+# To build Python C/C++ extension on Windows, we need to link to python import
library pythonXY.lib
+# See https://docs.python.org/3/extending/windows.html
+cc_import(
+ name="%{VARIETY_NAME}_lib",
+ interface_library=select({
+ "//:windows": ":%{VARIETY_NAME}_import_lib",
+ # A placeholder for Unix platforms which makes --no_build happy.
+ "//conditions:default": "not-existing.lib",
+ }),
+ system_provided=1,
+)
+
+cc_library(
+ name="%{VARIETY_NAME}_headers",
+ hdrs=[":%{VARIETY_NAME}_include"],
+ deps=select({
+ "//:windows": [":%{VARIETY_NAME}_lib"],
+ "//conditions:default": [],
+ }),
+ includes=["%{VARIETY_NAME}_include"],
+)
+
+%{PYTHON_INCLUDE_GENRULE}
+%{PYTHON_IMPORT_LIB_GENRULE}
diff --git a/cpp/fory/python/BUILD b/cpp/fory/python/BUILD
index 63f770a89..3197841f6 100644
--- a/cpp/fory/python/BUILD
+++ b/cpp/fory/python/BUILD
@@ -16,7 +16,7 @@
# under the License.
load("@rules_cc//cc:defs.bzl", "cc_library", "cc_test")
-load("@com_github_grpc_grpc//bazel:cython_library.bzl", "pyx_library")
+load("//bazel:cython_library.bzl", "pyx_library")
cc_library(
name = "_pyfory",
diff --git a/LICENSE b/licenses/LICENSE-tensorflow.txt
similarity index 70%
copy from LICENSE
copy to licenses/LICENSE-tensorflow.txt
index 0a90a1b47..12d255f8e 100644
--- a/LICENSE
+++ b/licenses/LICENSE-tensorflow.txt
@@ -200,108 +200,52 @@
See the License for the specific language governing permissions and
limitations under the License.
-================================================================
-
-This product includes a number of Dependencies with separate copyright notices
-and license terms. Your use of these submodules is subject to the terms and
-conditions of the following licenses.
-
-================================================================
-
-================================================================
-Apache-2.0 licenses
-================================================================
-The following components are provided under the Apache-2.0 License. See
project link for details.
-The text of each license is the standard Apache 2.0 license.
-
-* guava (https://github.com/google/guava)
- Files:
- java/fory-core/src/main/java/org/apache/fory/util/Preconditions.java
- java/fory-core/src/main/java/org/apache/fory/reflect/TypeParameter.java
- java/fory-core/src/main/java/org/apache/fory/reflect/TypeRef.java
-
java/fory-core/src/main/java/org/apache/fory/util/concurrency/DirectExecutorService.java
-
-* spark (https://github.com/apache/spark)
- Files:
- java/fory-core/src/main/java/org/apache/fory/codegen/Code.java
- java/fory-core/src/main/java/org/apache/fory/memory/Platform.java
-
java/fory-format/src/main/java/org/apache/fory/format/vectorized/ArrowWriter.java
-
-* commons-io (https://github.com/apache/commons-io)
- Files:
-
java/fory-core/src/main/java/org/apache/fory/io/ClassLoaderObjectInputStream.java
-
-* common-lang3 (https://github.com/apache/commons-lang)
- Files:
- java/fory-core/src/main/java/org/apache/fory/util/StringUtils.java
-
-* fast-serialization (https://github.com/RuedigerMoeller/fast-serialization)
- Files:
-
java/benchmark/src/main/java/org/apache/fory/benchmark/Identity2IdMap.java
-
-* ray (https://github.com/ray-project/ray)
- Files:
- ci/format.sh
- bazel/fory_deps_setup.bzl
- bazel/grpc-cython-copts.patch
- bazel/grpc-python.patch
-
-* tensorflow (https://github.com/tensorflow/tensorflow)
- Files:
- bazel/arrow/pyarrow_configure.bzl
-
-
-================================================================
-BSD-3-Clause licenses
-================================================================
-The following components are provided under the BSD-3-Clause License. See
project link for details.
-The text of each license is also included in licenses/LICENSE-[project].txt.
-
-* kryo (https://github.com/EsotericSoftware/kryo)
- Files:
- java/benchmark/src/main/java/org/apache/fory/benchmark/data/Data.java
- java/benchmark/src/main/java/org/apache/fory/benchmark/data/Image.java
- java/benchmark/src/main/java/org/apache/fory/benchmark/data/Media.java
-
java/benchmark/src/main/java/org/apache/fory/benchmark/data/MediaContent.java
- java/benchmark/src/main/java/org/apache/fory/benchmark/data/Sample.java
-
java/benchmark/src/main/java/org/apache/fory/benchmark/state/BufferType.java
-
java/benchmark/src/main/java/org/apache/fory/benchmark/state/KryoState.java
-
java/fory-core/src/main/java/org/apache/fory/collection/ForyObjectMap.java
- java/fory-core/src/main/java/org/apache/fory/collection/IdentityMap.java
-
java/fory-core/src/main/java/org/apache/fory/collection/IdentityObjectIntMap.java
- java/fory-core/src/main/java/org/apache/fory/collection/LongMap.java
-
java/fory-core/src/main/java/org/apache/fory/collection/LongLongByteMap.java
- java/fory-core/src/main/java/org/apache/fory/collection/ObjectIntMap.java
- java/fory-core/src/main/java/org/apache/fory/type/Generics.java
- java/fory-core/src/test/java/org/apache/fory/type/GenericsTest.java
-
-================================================================
-MIT licenses
-================================================================
-The following components are provided under the MIT License. See project link
for details.
-The text of each license is also included in licenses/LICENSE-[project].txt
-
-* murmurHash3.js (https://github.com/karanlyons/murmurHash3.js)
- Files:
- javascript/packages/fory/lib/murmurHash3.ts
-
-* cython-abseil-sample (https://github.com/messense/cython-abseil-sample)
- Files:
- python/pyfory/includes/libabsl.pxd
-
-
-================================================================
-Public Domain
-================================================================
-The following components are placed in the public domain.
-The author hereby disclaims copyright to this source code.
-See project link for details.
-
-* smhasher (https://github.com/aappleby/smhasher)
- Files:
- src/fory/thirdparty/MurmurHash3.cc
- src/fory/thirdparty/MurmurHash3.h
-
-* java_util (https://github.com/yonik/java_util)
- Files:
- java/fory-core/src/main/java/org/apache/fory/util/MurmurHash3.java
+## Some of TensorFlow's code is derived from Caffe, which is subject to the
following copyright notice:
+
+COPYRIGHT
+
+All contributions by the University of California:
+
+Copyright (c) 2014, The Regents of the University of California (Regents)
+All rights reserved.
+
+All other contributions:
+
+Copyright (c) 2014, the respective contributors
+All rights reserved.
+
+Caffe uses a shared copyright model: each contributor holds copyright over
+their contributions to Caffe. The project versioning records all such
+contribution and copyright details. If a contributor wants to further mark
+their specific copyright on a particular contribution, they should indicate
+their copyright solely in the commit message of the change when it is
+committed.
+
+LICENSE
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND
+ ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED
+ WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
FOR
+ ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES
+ (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS
+ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+CONTRIBUTION AGREEMENT
+
+By contributing to the BVLC/caffe repository through pull-request, comment,
+or otherwise, the contributor releases their content to the
+license and copyright terms herein.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
