This is an automated email from the ASF dual-hosted git repository.
janardhan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/systemml.git
The following commit(s) were added to refs/heads/master by this push:
new dbe00f1 [MINOR][BUILD] Package python dist. with mvn `-P` flag
dbe00f1 is described below
commit dbe00f140a63329e26f269890828679cd81f2a97
Author: Kevin Innerebner <[email protected]>
AuthorDate: Wed Apr 22 20:33:33 2020 +0530
[MINOR][BUILD] Package python dist. with mvn `-P` flag
Maven build
- Maven packaging with new `-P distribution` option
- New pypi release instructions and pre_setup.py
Testing
- Run all tests with single command in workflow
Doc
- Updates the setup.py, to read `README.md` to remove
redundancy
Closes #873.
---
.github/workflows/python.yml | 15 ++-------
src/main/python/BUILD_INSTRUCTIONS.md | 56 ---------------------------------
src/main/python/PUBLISH_INSTRUCTIONS.md | 53 +++++++++++++++++++++++++++++++
src/main/python/README.md | 22 ++++++++++++-
src/main/python/docs/source/install.rst | 2 +-
src/main/python/pre_setup.py | 44 +++++++++++++++++++-------
src/main/python/setup.py | 18 ++---------
7 files changed, 113 insertions(+), 97 deletions(-)
diff --git a/.github/workflows/python.yml b/.github/workflows/python.yml
index 3c6a952..cc3e1cb 100644
--- a/.github/workflows/python.yml
+++ b/.github/workflows/python.yml
@@ -57,7 +57,7 @@ jobs:
${{ runner.os }}-maven-
- name: Maven clean & package
- run: mvn clean package
+ run: mvn clean package -P distribution
- name: Setup Python
uses: actions/setup-python@v1
@@ -86,16 +86,5 @@ jobs:
cd src/main/python
ls tests/
echo "Beginning tests"
- python tests/test_matrix_binary_op.py
- echo "Exit Status: " $?
- sleep 3
- python tests/test_matrix_aggregations.py
- echo "Exit Status: " $?
- sleep 3
- python tests/test_l2svm.py
- echo "Exit Status: " $?
- python tests/test_lineagetrace.py
- sleep 3
- echo "Exit Status: " $?
- python tests/test_l2svm_lineage.py
+ python -m unittest tests/*.py
echo "Exit Status: " $?
diff --git a/src/main/python/BUILD_INSTRUCTIONS.md
b/src/main/python/BUILD_INSTRUCTIONS.md
deleted file mode 100644
index dd6adb8..0000000
--- a/src/main/python/BUILD_INSTRUCTIONS.md
+++ /dev/null
@@ -1,56 +0,0 @@
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% end comment %}
--->
-
-# Build instructions
-
-## Basic steps
-
-The following steps have to be done for both cases
-
-- Build SystemDS with maven first `mvn package -DskipTests`, with the working
directory being `SYSTEMDS_ROOT` (Root directory of SystemDS)
-- `cd` to this folder (basically `SYSTEMDS_ROOT/src/main/python`
-
-### Building package
-
-If we want to build the package for uploading to the repository via `python3
-m twine upload --repository-url [URL] dist/*` (will be automated in the future)
-
-- Run `create_python_dist.py`
-
-```bash
-python3 create_python_dist.py
-```
-
-- now in the `./dist` directory there will exist the source distribution
`systemds-VERSION.tar.gz` and the wheel distribution
`systemds-VERSION-py3-none-any.whl`, with `VERSION` being the current version
number
-- Finished. We can now upload it with `python3 -m twine upload
--repository-url [URL] dist/*`
-
-### Building for development
-
-If we want to build the package just locally for development, the following
steps will suffice
-
-- Run `pre_setup.py` (this will copy `lib` and `systemds-VERSION-SNAPSHOT.jar`)
-
-```bash
-python3 create_python_dist.py
-```
-
-- Finished. Test by running a test case of your choice:
-
-```bash
-python3 tests/test_matrix_binary_op.py
-```
diff --git a/src/main/python/PUBLISH_INSTRUCTIONS.md
b/src/main/python/PUBLISH_INSTRUCTIONS.md
new file mode 100644
index 0000000..bc5ce32
--- /dev/null
+++ b/src/main/python/PUBLISH_INSTRUCTIONS.md
@@ -0,0 +1,53 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% end comment %}
+-->
+
+# Publishing Instructions
+
+## Building SystemDS jar (with dependency jars)
+
+The following steps have to be done for both the cases
+
+- Build SystemDS with maven first `mvn package -P distribution`, with the
working
+ directory being `SYSTEMDS_ROOT` (Root directory of SystemDS)
+- `cd` to this folder (basically `SYSTEMDS_ROOT/src/main/python`)
+
+## Building python package
+
+- Run `create_python_dist.py`
+
+```bash
+python3 create_python_dist.py
+```
+
+- now in the `./dist` directory there will exist the source distribution
`systemds-VERSION.tar.gz`
+ and the wheel distribution `systemds-VERSION-py3-none-any.whl`, with
`VERSION` being the current version number
+
+## Publishing package
+
+If we want to build the package for uploading to the repository via `python3
-m twine upload dist/*`
+ (will be automated in the future)
+
+- Install twine with `pip install --upgrade twine`
+
+- Follow the instructions from the
[Guide](https://packaging.python.org/tutorials/packaging-projects/)
+ 1. Create an API-Token in the account (leave the page open or copy the
token, it will only be shown once)
+ 2. Execute the command `python3 -m twine upload dist/*`
+ - Optional: `pip install keyrings.alt`(use with caution!) if you get
`UserWarning: No recommended backend was available.`
+ 3. Username is `__token__`
+ 4. Password is the created API-Token **with** `pypi-` prefix
diff --git a/src/main/python/README.md b/src/main/python/README.md
index 1df8660..2850b11 100644
--- a/src/main/python/README.md
+++ b/src/main/python/README.md
@@ -1,4 +1,24 @@
-# Python Readme
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% end comment %}
+-->
+# SystemDS
+
+
This package provides a Pythonic interface for working with SystemDS.
diff --git a/src/main/python/docs/source/install.rst
b/src/main/python/docs/source/install.rst
index f200304..a667df2 100644
--- a/src/main/python/docs/source/install.rst
+++ b/src/main/python/docs/source/install.rst
@@ -68,7 +68,7 @@ Then to build the system you do the following
- Clone the Git Repository: https://github.com/apache/systemml.git
- Open an terminal at the root of the repository.
-- Package the Java code using the ``mvn package`` command
+- Package the Java code using the ``mvn package -P distribution`` command
- ``cd src/main/python`` to point at the root of the SystemDS Python library.
- Copy `jars` with ``python pre_setup.py``
- Install with ``pip install .``
diff --git a/src/main/python/pre_setup.py b/src/main/python/pre_setup.py
index 6925864..114260c 100755
--- a/src/main/python/pre_setup.py
+++ b/src/main/python/pre_setup.py
@@ -23,24 +23,46 @@
import os
import shutil
import fnmatch
+from zipfile import ZipFile
+this_path = os.path.dirname(os.path.realpath(__file__))
python_dir = 'systemds'
java_dir = 'systemds-java'
-java_dir_full_path = os.path.join(python_dir, java_dir)
+java_dir_full_path = os.path.join(this_path, python_dir, java_dir)
if os.path.exists(java_dir_full_path):
shutil.rmtree(java_dir_full_path, True)
-os.mkdir(java_dir_full_path)
-root_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.getcwd())))
-JAR_FILE_NAMES = ['systemds-*-SNAPSHOT.jar', 'systemds-*.jar',
'systemds-*-SNAPSHOT-extra', 'systemds-*-extra.jar']
-EXCLUDE_JAR_FILE_NAMES = ['systemds-*javadoc.jar', 'systemds-*sources.jar',
'systemds-*standalone.jar',
- 'systemds-*lite.jar']
+root_dir = os.path.dirname(os.path.dirname(os.path.dirname(this_path)))
+
+# temporary directory for unzipping of bin zip
+TMP_DIR = os.path.join(this_path, 'tmp')
+if os.path.exists(TMP_DIR):
+ shutil.rmtree(TMP_DIR, True)
+os.mkdir(TMP_DIR)
+
+SYSTEMDS_BIN = 'systemds-*-SNAPSHOT-bin.zip'
for file in os.listdir(os.path.join(root_dir, 'target')):
- if any((fnmatch.fnmatch(file, valid_name) for valid_name in
JAR_FILE_NAMES)) and not any(
- (fnmatch.fnmatch(file, exclude_name) for exclude_name in
EXCLUDE_JAR_FILE_NAMES)):
- shutil.copyfile(os.path.join(root_dir, 'target', file),
os.path.join(java_dir_full_path, file))
- if file == 'lib':
- shutil.copytree(os.path.join(root_dir, 'target', file),
os.path.join(java_dir_full_path, file))
+ if fnmatch.fnmatch(file, SYSTEMDS_BIN):
+ new_path = os.path.join(TMP_DIR, file)
+ shutil.copyfile(os.path.join(root_dir, 'target', file), new_path)
+ extract_dir = os.path.join(TMP_DIR)
+ with ZipFile(new_path, 'r') as zip:
+ for f in zip.namelist():
+ split_path = os.path.split(os.path.dirname(f))
+ if split_path[1] == 'lib':
+ zip.extract(f, TMP_DIR)
+ unzipped_dir_name = file.rsplit('.', 1)[0]
+ shutil.copytree(os.path.join(TMP_DIR, unzipped_dir_name),
java_dir_full_path)
+ if os.path.exists(TMP_DIR):
+ shutil.rmtree(TMP_DIR, True)
root_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.getcwd())))
shutil.copyfile(os.path.join(root_dir, 'LICENSE'), 'LICENSE')
shutil.copyfile(os.path.join(root_dir, 'NOTICE'), 'NOTICE')
+
+# delete old build and dist path
+build_path = os.path.join(this_path, 'build')
+if os.path.exists(build_path):
+ shutil.rmtree(build_path, True)
+dist_path = os.path.join(this_path, 'dist')
+if os.path.exists(dist_path):
+ shutil.rmtree(dist_path, True)
diff --git a/src/main/python/setup.py b/src/main/python/setup.py
index 3fece95..27b3af9 100755
--- a/src/main/python/setup.py
+++ b/src/main/python/setup.py
@@ -20,7 +20,6 @@
#
#-------------------------------------------------------------
-from __future__ import print_function
import sys
from setuptools import find_packages, setup
import time
@@ -37,7 +36,7 @@ ARTIFACT_VERSION = __project_version__
ARTIFACT_VERSION_SHORT = ARTIFACT_VERSION.split("-")[0]
REQUIRED_PACKAGES = [
- 'numpy >= 1.8.2', # we might want to recheck this version
+ 'numpy >= 1.8.2',
'py4j >= 0.10.0'
]
@@ -49,19 +48,7 @@ setup(
name=ARTIFACT_NAME,
version=ARTIFACT_VERSION_SHORT,
description='SystemDS is a distributed and declarative machine learning
platform.',
- long_description='''SystemDS is a versatile system for the end-to-end data
science lifecycle from data integration,
- cleaning, and feature engineering, over efficient, local and distributed
ML model training,
- to deployment and serving.
- To facilitate this, bindings from different languages and different
system abstractions provide help for:
- (1) the different tasks of the data-science lifecycle, and
- (2) users with different expertise.
-
-
- These high-level scripts are compiled into hybrid execution plans of
local, in-memory CPU and GPU operations,
- as well as distributed operations on Apache Spark. In contrast to
existing systems
- - that either provide homogeneous tensors or 2D Datasets - and in order
to serve the entire
- data science lifecycle, the underlying data model are DataTensors, i.e.,
- tensors (multi-dimensional arrays) whose first dimension may have a
heterogeneous and nested schema.''',
+ long_description=open('README.md').read(),
url='https://github.com/apache/systemml',
author='SystemDS',
author_email='[email protected]',
@@ -79,6 +66,7 @@ setup(
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
+ 'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3 :: Only',
'Topic :: Scientific/Engineering :: Mathematics',
'Topic :: Software Development :: Libraries :: Python Modules',
