This is an automated email from the ASF dual-hosted git repository.
yaniv pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-amaterasu.git.
from b0b6472 Merge pull request #49 from roadan/AMATERASU-69
add 5a27029 first version of sdk + minimal tests
add e4ad874 rebase in work
add 80ae397 base datastores API + tests
add dc12289 spark datastores integration.
add 54edcc3 license
add 9e10056 names better describe the nature of the beast
add 5868108 renaming leftovers
add 2b410f1 version bump
add 78f5876 python requirements now better handled
add 2466cdb Configuration fixes, fixed some issues that resulted in NPEs
across teh board Mesos native lib path is now configurable also a bug that
happens if Amaterasu is stopped during the run would make the repo dir stay
after exit. Now deleted on every new launch.
add e6d117a Incorrect location where ama-requirements.txt was generated
add d79baae more resources resolution issues
add 3e3d517 reverted version bump
add 37e0ec6 and more resources issues
add 5ab92ea command returns is empty
add 9c2c87d command returns empty for some reason
add aee2541 command wasn't generated
add 2f5ae2c command wasn't generated
add 2c85d69 command concat error
add 1147dde assumed pip to be installed
add c09a303 copy action source file to container
add 1e19ccf need to overwrite, otherwise it fails on retry
add da458e8 incorrect path passed to container
add 51b004e incorrect path in command
add 346fe82 builder architecture
add 292a321 added stubs for spark
add be8dbf9 typo
add c96fe83 in case where the runtime isn't completely isolated (e.g. -
mesos native containers), the python dependencies could end up not updating in
case of a python sdk or spark update.
add 7d180d0 still some issues with resolution of dependencies
add 8a3bc98 removed obsolete code
add 2acc932 removed obsolete code
add 148cad9 when working through spark-submit, unless something changed,
can use the default sparkcontext and session
add f917f05 default conf refitted to use default conf files form leader
add f810191 default conf refitted to use default conf files form leader
add 3dd3571 default conf refitted to use default conf files form leader
add 8f0bc6d wrong name from leader
add 36fe1bd env not loaded correctly
add f60183e env not loaded correctly
add 9ced5d5 env not loaded correctly
add a92abeb env not loaded correctly
add bd0044c set pyspark_python
add 79e06ea set pyspark_python
add 88309c3 set pyspark_python
add 8e12f48 trying to get worker to use the correct python executable
add 717c488 trying to get worker to use the correct python executable
add ab4b636 trying to get worker to use the correct python executable
add e5ea38f trying to get worker to use the correct python executable
add a98a45b trying to get worker to use the correct python executable
add b1ec3c1 pyspark works but the executors terminates for no apparent
reason
add ece1ee2 trying to notify the master that the task finished
successfully
add 374ba97 trying to notify the master that the task finished
successfully
add e6e1fcd duh
add 9fdd3b4 duh2
add 7ad4428 still figuring out what's going on
add 8cce4c1 still figuring out what's going on
add 283bf62 still figuring out what's going on
add fb73685 still figuring out what's going on
add 87614be still figuring out what's going on
add c12a270 still figuring out what's going on
add c3717f3 still trying to acknoledge task from within the executor
add d1ecc46 still figuring out what's going on
add 3fdcc18 still trying to acknoledge task from within the executor
add 114ebba still trying to acknoledge task from within the executor
add e285995 still trying to acknoledge task from within the executor
add 44695de pyspark works. Started basic python runner.
add ef483e9 added support for pandas, fixed support for basic python
add 075b73a merged and tested on yarn
add aff8b49 This actually caused the total destruction of any previously
defined user resources
add 09a8393 added support for basic python and pandas, pandas not fully
integrated.
add c474d54 fixed merge
add 812d36a python path is configurable now
add c66e98d removed platform specific command resolution
add 40600a4 fixing pip install
add e35fa6c fixing pip install
add aea7f6d changed to use a localized virtualenv
new b53d7c0 fixed the spark-env location
new 3a872be virtual env is now available
new cdaaf51 fixed the spark-env location
new f9398c9 fixed the spark-env location
new 72668dc incorrect command concatenation
new fbbf222 Merge branch 'nadav-har-tzvi-AMATERASU-45-Rebase' of
github.com:nadav-har-tzvi/incubator-amaterasu into
nadav-har-tzvi-AMATERASU-45-Rebase
new 74e24bf fixed YARN java encoding
new 0543dae datasets.ymal is available for yarn
new 02c3d87 testing requeued actions
new 6722397 updated yarn version
new a39bcf7 fixed yarm AM memory issue
new 2b05d2e updated curator version
new 4681bbb updated curator version
new 3b42585 handling container allocation issues
new a5db07f setting deploy mode
new e7715eb setting deploy mode
new 0044204 setting deploy mode
new d7ad642 fixed memory issues in AM
new 0242b98 fixing hive-site
new 8943ba1 fixing hive-site
new 667bb98 fixing virtual env config
new 81ff18e fixing virtual env config
new dce8de4 fixing virtual env config
new a1b78bb spark_conf pulled from spark-submit conf if exists
new 886541c Installing action requirements inside the driver and shipping
them to the executors
new 2e7904d fixing master
new 59c4c1c Installing action requirements inside the driver and shipping
them to the executors
new 8043d0e virtualenv isn't supported by spark, provided alternatives
new cefc2b0 duh
new 36e219e disabled the activeMQ logger for now
new 4691bb0 creating datasets.yaml in mesos
new 9d14164 mesos executing the new python support
new a28eea9 mesos notifications added
new d39a4c4 fixing ama-requierments location in YARN
new f8b8fc4 copy fixed sofr mesos
new 4f48c70 Merge branch 'nadav-har-tzvi-AMATERASU-45-Rebase'
The 36 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
common/build.gradle | 9 +-
.../execution/dependencies/PythonDependencies.kt | 2 +-
.../common/configuration/ClusterConfig.scala | 33 +++-
.../common/configuration/SparkConfig.scala | 2 +-
.../python/dispatcher}/build.gradle | 50 +++---
.../python/dispatcher/PythonSetupProvider.kt | 53 ++++++
.../runners/providers/BasicPythonRunnerProvider.kt | 36 ++--
.../runners/providers/PandasRunnerProvider.kt | 20 +++
.../runners/providers/PythonRunnerProviderBase.kt | 76 +++++++++
.../dispatcher/BasicPythonRunnerProviderTests.kt | 88 ++++++++++
frameworks/python/pandas_runtime/__init__.py | 16 ++
.../amaterasu_pandas.egg-info/PKG-INFO | 28 +++
.../amaterasu_pandas.egg-info/SOURCES.txt | 11 ++
.../amaterasu_pandas.egg-info/dependency_links.txt | 1 +
.../amaterasu_pandas.egg-info/entry_points.txt | 3 +
.../amaterasu_pandas.egg-info/requires.txt | 8 +
.../amaterasu_pandas.egg-info/top_level.txt | 2 +
frameworks/python/pandas_runtime/build.gradle | 13 ++
.../dist/amaterasu_pandas-0.2.0-incubating-rc4.zip | Bin 0 -> 8304 bytes
.../pandas_runtime/pandas_runtime/__init__.py | 16 ++
.../pandas_runtime/pandas_runtime/datasets.py | 134 +++++++++++++++
.../pandas_runtime/pandas_runtime/runtime.py | 35 ++++
frameworks/python/pandas_runtime/setup.py | 51 ++++++
.../pandas_runtime/tests/__init__.py} | 0
frameworks/python/python_runtime/__init__.py | 16 ++
.../amaterasu_python.egg-info/PKG-INFO | 28 +++
.../amaterasu_python.egg-info/SOURCES.txt | 9 +
.../amaterasu_python.egg-info/dependency_links.txt | 1 +
.../amaterasu_python.egg-info/entry_points.txt | 3 +
.../amaterasu_python.egg-info/requires.txt | 1 +
.../amaterasu_python.egg-info/top_level.txt | 1 +
frameworks/python/python_runtime/build.gradle | 13 ++
.../dist/amaterasu_python-0.2.0-incubating-rc4.zip | Bin 0 -> 6167 bytes
.../python_runtime/python_runtime/__init__.py | 16 ++
.../python_runtime/python_runtime/runtime.py | 33 ++++
frameworks/python/python_runtime/setup.py | 51 ++++++
frameworks/spark/dispatcher/build.gradle | 28 ++-
.../spark/dispatcher/SparkSetupProvider.kt | 103 +++++++++++
.../runners/providers/PySparkRunnerProvider.kt | 49 ++++++
.../providers/SparkSubmitScalaRunnerProvider.kt | 44 +++++
.../spark/dispatcher/SparkSetupProvider.scala | 119 -------------
.../runners/providers/PySparkRunnerProvider.scala | 54 ------
.../providers/SparkScalaRunnerProvider.scala | 73 --------
.../providers/SparkSubmitScalaRunnerProvider.scala | 49 ------
frameworks/spark/pyspark_runtime/__init__.py | 16 ++
.../amaterasu_pyspark.egg-info/PKG-INFO | 28 +++
.../amaterasu_pyspark.egg-info/SOURCES.txt | 16 ++
.../dependency_links.txt | 1 +
.../amaterasu_pyspark.egg-info/entry_points.txt | 3 +
.../amaterasu_pyspark.egg-info/requires.txt | 2 +
.../amaterasu_pyspark.egg-info/top_level.txt | 2 +
.../pyspark_runtime/amaterasu_pyspark/__init__.py | 16 ++
.../pyspark_runtime/amaterasu_pyspark/datasets.py | 73 ++++++++
.../pyspark_runtime/amaterasu_pyspark/runtime.py | 84 +++++++++
frameworks/spark/pyspark_runtime/build.gradle | 13 ++
.../amaterasu_pyspark-0.2.0-incubating-rc4.zip | Bin 0 -> 14488 bytes
frameworks/spark/pyspark_runtime/setup.py | 51 ++++++
frameworks/spark/pyspark_runtime/tests/__init__.py | 16 ++
frameworks/spark/pyspark_runtime/tests/base.py | 51 ++++++
.../spark/pyspark_runtime/tests/datasets.yml | 19 ++-
.../spark/pyspark_runtime/tests/env.yml | 20 ++-
.../pyspark_runtime/tests/integration_tests.py | 83 +++++++++
.../spark/pyspark_runtime/tests/runtime.yml | 7 +-
.../spark/pyspark_runtime/tests/runtime_tests.py | 85 ++++++++++
frameworks/spark/pyspark_runtime/tests/sample.py | 24 +++
.../pyspark_runtime/tests/test_scripts/simple.py | 36 ++++
.../spark/pyspark_runtime/tests/unit_tests.py | 62 +++++++
.../spark/runner/SparkRunnersProvider.scala | 42 ++---
.../spark/runner/pyspark/PySparkRunner.scala | 4 -
.../spark/runner/repl/SparkRunnerHelper.scala | 6 +-
.../frameworks/spark/runner/SparkTestsSuite.scala | 2 +-
leader-common/build.gradle | 8 +-
.../apache/amaterasu/leader/common/dsl/GitUtil.kt | 2 +
.../leader/common/execution/JobManager.kt | 2 +-
.../frameworls/FrameworkProvidersFactory.kt | 1 -
.../leader/common/utilities/DataLoader.kt | 29 +++-
.../amaterasu/leader/common/dsl/GitUtilTests.kt | 18 ++
.../leader/common/dsl/JobParserArtifactTests.kt | 4 +-
leader-yarn/build.gradle | 3 +-
.../amaterasu/leader/yarn/ApplicationMaster.kt | 47 ++++--
.../org/apache/amaterasu/leader/yarn/Client.kt | 15 +-
.../amaterasu/leader/yarn/YarnNMCallbackHandler.kt | 16 +-
leader/build.gradle | 5 +-
.../amaterasu/leader/mesos/MesosJobLauncher.scala | 1 -
.../leader/mesos/schedulers/JobScheduler.scala | 85 ++++++----
leader/src/main/scripts/ama-start-mesos.sh | 19 ++-
leader/src/main/scripts/ama-start-yarn.sh | 8 +-
leader/src/main/scripts/amaterasu.properties | 6 +-
leader/src/test/resources/amaterasu.properties | 25 +--
.../common/execution/JobExecutionTests.scala | 17 ++
sdk/build.gradle | 10 +-
.../sdk/frameworks/RunnerSetupProvider.kt | 12 +-
sdk_python/__init__.py | 16 ++
sdk_python/amaterasu/__init__.py | 16 ++
sdk_python/amaterasu/base/__init__.py | 2 +
sdk_python/amaterasu/base/datasets.py | 116 +++++++++++++
sdk_python/amaterasu/base/runtime.py | 188 +++++++++++++++++++++
.../amaterasu/pandas/__init__.py | 0
sdk_python/amaterasu/pandas/datasets.py | 134 +++++++++++++++
sdk_python/amaterasu/pandas/runtime.py | 35 ++++
.../amaterasu/pyspark/__init__.py | 0
sdk_python/amaterasu/pyspark/datasets.py | 73 ++++++++
sdk_python/amaterasu/pyspark/runtime.py | 91 ++++++++++
sdk_python/amaterasu_sdk.egg-info/PKG-INFO | 28 +++
sdk_python/amaterasu_sdk.egg-info/SOURCES.txt | 16 ++
.../amaterasu_sdk.egg-info/dependency_links.txt | 1 +
sdk_python/amaterasu_sdk.egg-info/requires.txt | 3 +
sdk_python/amaterasu_sdk.egg-info/top_level.txt | 1 +
sdk_python/build.gradle | 13 ++
.../dist/amaterasu-sdk-0.2.0-incubating-rc4.zip | Bin 0 -> 15020 bytes
sdk_python/setup.py | 60 +++++++
settings.gradle | 21 +++
112 files changed, 2776 insertions(+), 511 deletions(-)
copy {leader-yarn => frameworks/python/dispatcher}/build.gradle (79%)
create mode 100644
frameworks/python/dispatcher/src/main/kotlin/org/apache/amaterasu/frameworks/python/dispatcher/PythonSetupProvider.kt
copy
leader-common/src/main/kotlin/org/apache/amaterasu/leader/common/execution/frameworls/FrameworkProvidersFactory.kt
=>
frameworks/python/dispatcher/src/main/kotlin/org/apache/amaterasu/frameworks/python/dispatcher/runners/providers/BasicPythonRunnerProvider.kt
(50%)
create mode 100644
frameworks/python/dispatcher/src/main/kotlin/org/apache/amaterasu/frameworks/python/dispatcher/runners/providers/PandasRunnerProvider.kt
create mode 100644
frameworks/python/dispatcher/src/main/kotlin/org/apache/amaterasu/frameworks/python/dispatcher/runners/providers/PythonRunnerProviderBase.kt
create mode 100644
frameworks/python/dispatcher/src/test/kotlin/org/apache/amaterasu/frameworks/python/dispatcher/BasicPythonRunnerProviderTests.kt
create mode 100644 frameworks/python/pandas_runtime/__init__.py
create mode 100644
frameworks/python/pandas_runtime/amaterasu_pandas.egg-info/PKG-INFO
create mode 100644
frameworks/python/pandas_runtime/amaterasu_pandas.egg-info/SOURCES.txt
create mode 100644
frameworks/python/pandas_runtime/amaterasu_pandas.egg-info/dependency_links.txt
create mode 100644
frameworks/python/pandas_runtime/amaterasu_pandas.egg-info/entry_points.txt
create mode 100644
frameworks/python/pandas_runtime/amaterasu_pandas.egg-info/requires.txt
create mode 100644
frameworks/python/pandas_runtime/amaterasu_pandas.egg-info/top_level.txt
create mode 100644 frameworks/python/pandas_runtime/build.gradle
create mode 100644
frameworks/python/pandas_runtime/dist/amaterasu_pandas-0.2.0-incubating-rc4.zip
create mode 100644 frameworks/python/pandas_runtime/pandas_runtime/__init__.py
create mode 100644 frameworks/python/pandas_runtime/pandas_runtime/datasets.py
create mode 100644 frameworks/python/pandas_runtime/pandas_runtime/runtime.py
create mode 100644 frameworks/python/pandas_runtime/setup.py
copy frameworks/{spark/runner/src/test/resources/SparkSql/parquet/_SUCCESS =>
python/pandas_runtime/tests/__init__.py} (100%)
create mode 100644 frameworks/python/python_runtime/__init__.py
create mode 100644
frameworks/python/python_runtime/amaterasu_python.egg-info/PKG-INFO
create mode 100644
frameworks/python/python_runtime/amaterasu_python.egg-info/SOURCES.txt
create mode 100644
frameworks/python/python_runtime/amaterasu_python.egg-info/dependency_links.txt
create mode 100644
frameworks/python/python_runtime/amaterasu_python.egg-info/entry_points.txt
create mode 100644
frameworks/python/python_runtime/amaterasu_python.egg-info/requires.txt
create mode 100644
frameworks/python/python_runtime/amaterasu_python.egg-info/top_level.txt
create mode 100644 frameworks/python/python_runtime/build.gradle
create mode 100644
frameworks/python/python_runtime/dist/amaterasu_python-0.2.0-incubating-rc4.zip
create mode 100644 frameworks/python/python_runtime/python_runtime/__init__.py
create mode 100644 frameworks/python/python_runtime/python_runtime/runtime.py
create mode 100644 frameworks/python/python_runtime/setup.py
create mode 100644
frameworks/spark/dispatcher/src/main/kotlin/org/apache/amaterasu/frameworks/spark/dispatcher/SparkSetupProvider.kt
create mode 100644
frameworks/spark/dispatcher/src/main/kotlin/org/apache/amaterasu/frameworks/spark/dispatcher/runners/providers/PySparkRunnerProvider.kt
create mode 100644
frameworks/spark/dispatcher/src/main/kotlin/org/apache/amaterasu/frameworks/spark/dispatcher/runners/providers/SparkSubmitScalaRunnerProvider.kt
delete mode 100644
frameworks/spark/dispatcher/src/main/scala/org/apache/amaterasu/frameworks/spark/dispatcher/SparkSetupProvider.scala
delete mode 100644
frameworks/spark/dispatcher/src/main/scala/org/apache/amaterasu/frameworks/spark/dispatcher/runners/providers/PySparkRunnerProvider.scala
delete mode 100644
frameworks/spark/dispatcher/src/main/scala/org/apache/amaterasu/frameworks/spark/dispatcher/runners/providers/SparkScalaRunnerProvider.scala
delete mode 100644
frameworks/spark/dispatcher/src/main/scala/org/apache/amaterasu/frameworks/spark/dispatcher/runners/providers/SparkSubmitScalaRunnerProvider.scala
create mode 100644 frameworks/spark/pyspark_runtime/__init__.py
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark.egg-info/PKG-INFO
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark.egg-info/SOURCES.txt
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark.egg-info/dependency_links.txt
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark.egg-info/entry_points.txt
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark.egg-info/requires.txt
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark.egg-info/top_level.txt
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark/__init__.py
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark/datasets.py
create mode 100644
frameworks/spark/pyspark_runtime/amaterasu_pyspark/runtime.py
create mode 100644 frameworks/spark/pyspark_runtime/build.gradle
create mode 100644
frameworks/spark/pyspark_runtime/dist/amaterasu_pyspark-0.2.0-incubating-rc4.zip
create mode 100644 frameworks/spark/pyspark_runtime/setup.py
create mode 100644 frameworks/spark/pyspark_runtime/tests/__init__.py
create mode 100644 frameworks/spark/pyspark_runtime/tests/base.py
copy leader-common/src/test/resources/test_repo/src/start/env/test/job.yml =>
frameworks/spark/pyspark_runtime/tests/datasets.yml (79%)
copy leader-common/src/test/resources/test_repo/env/test/job.yml =>
frameworks/spark/pyspark_runtime/tests/env.yml (66%)
create mode 100644 frameworks/spark/pyspark_runtime/tests/integration_tests.py
copy leader-common/src/test/resources/test_repo/src/start/env/test/job.yml =>
frameworks/spark/pyspark_runtime/tests/runtime.yml (91%)
create mode 100644 frameworks/spark/pyspark_runtime/tests/runtime_tests.py
create mode 100644 frameworks/spark/pyspark_runtime/tests/sample.py
create mode 100644
frameworks/spark/pyspark_runtime/tests/test_scripts/simple.py
create mode 100644 frameworks/spark/pyspark_runtime/tests/unit_tests.py
create mode 100644
leader-common/src/test/kotlin/org/apache/amaterasu/leader/common/dsl/GitUtilTests.kt
create mode 100644 sdk_python/__init__.py
create mode 100644 sdk_python/amaterasu/__init__.py
create mode 100644 sdk_python/amaterasu/base/__init__.py
create mode 100644 sdk_python/amaterasu/base/datasets.py
create mode 100644 sdk_python/amaterasu/base/runtime.py
copy frameworks/spark/runner/src/test/resources/SparkSql/parquet/_SUCCESS =>
sdk_python/amaterasu/pandas/__init__.py (100%)
create mode 100644 sdk_python/amaterasu/pandas/datasets.py
create mode 100644 sdk_python/amaterasu/pandas/runtime.py
copy frameworks/spark/runner/src/test/resources/SparkSql/parquet/_SUCCESS =>
sdk_python/amaterasu/pyspark/__init__.py (100%)
create mode 100644 sdk_python/amaterasu/pyspark/datasets.py
create mode 100644 sdk_python/amaterasu/pyspark/runtime.py
create mode 100644 sdk_python/amaterasu_sdk.egg-info/PKG-INFO
create mode 100644 sdk_python/amaterasu_sdk.egg-info/SOURCES.txt
create mode 100644 sdk_python/amaterasu_sdk.egg-info/dependency_links.txt
create mode 100644 sdk_python/amaterasu_sdk.egg-info/requires.txt
create mode 100644 sdk_python/amaterasu_sdk.egg-info/top_level.txt
create mode 100644 sdk_python/build.gradle
create mode 100644 sdk_python/dist/amaterasu-sdk-0.2.0-incubating-rc4.zip
create mode 100644 sdk_python/setup.py
