This is an automated email from the ASF dual-hosted git repository.
zkaoudi pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-wayang.git
from 93f2d839 Merge pull request #398 from juripetersen/ml-model-guide
add 2192b8d7 removed github actions
add be2572d9 removed github actions
add b884e769 removed ASF license
add ef213dd8 Update README.md
add a569d116 Merge pull request #2 from databloom-ai/2pk03-patch-1
add 70afca9a Update README.md
add d8c486b3 Merge pull request #3 from databloom-ai/2pk03-patch-1
add 34b93e82 Update README.md
add 99dd0ed7 Merge pull request #4 from databloom-ai/2pk03-patch-1
add 5834072e [wayang-platforms] Added sql to uncached rdd channel
conversion.
add 93f466ca [wayang-platforms] Small fix for previous commit.
add 301c2ed4 [wayang-platforms] Small tweak for SqlToRddOperator.
add d1f81546 [wayang-api-scala-java] Added initial implementation of
MultiContextPlanBuilder.scala
add 079831c4 [wayang-api-scala-java] MultiContextPlanBuilder: Added
withUdfJars method
add c1cf1871 [wayang-api-scala-java] MultiContextPlanBuilder: A sink is
now bound to a context. Also added a thread per context.
add eca74468 [wayang-api-scala-java] Working on plan serialization
add 87383fd2 [wayang-api-scala-java] Still working on plan serialization
add 991cba1c [wayang-api-scala-java] First working version of plan
serialization
add 88e6ff2c [wayang-api-scala-java] [multi-context-plan-builder] Working
on testing and benchmarks
add bedbe4ef [wayang-api-scala-java] [multi-context-plan-builder] More
operators can be serialized & more tests
add 18153f91 [wayang-api-scala-java] [multi-context-plan-builder] Added a
word count benchmark
add 3996df31 [wayang-api-scala-java] [multi-context-plan-builder] Some
tweaks for running on gcp
add d0242ef6 [wayang-api-scala-java] [multi-context-plan-builder] Added
binary and loop operators & tests
add 7c0ba491 [multi-context-plan-builder] Tweaks for loop and sample
operator serialization
add ba22ec47 [wayang-benchmark] [multi-context-plan-builder] Tweaked
WordCount and added TPCHQuery1
add f31a5535 [wayang-benchmark] [multi-context-plan-builder] TPCHQuery1:
Tweak
add c06afcd1 [multi-context-plan-builder] Added target platform
serialization
add bd663133 [multi-context-plan-builder] Added multiple sources, working
on target platforms
add 3a1ce062 [multi-context-plan-builder] Fixed target platforms
serialization
add 23552888 [multi-context-plan-builder] Refactored benchmarks
add 06d9ca44 [multi-context-plan-builder] Tweaks and cleanup
add 5812f83c [multi-context-plan-builder] Fixed source and output of join
and coGroup test
add 8dcbc92a [multi-context-plan-builder] Cleanup on SerializationUtils
add 02a1db9f [multi-context-plan-builder] Serialization/deserialization
fixes for table sources and connecting loop operators to other operators
add ff046559 [multi-context-plan-builder] Added merging of contexts
add b530ff65 [multi-context-plan-builder] Tweaks in mergeUnion
add 47511ab0 [multi-context-plan-builder] Working on async executions and
async combinations
add a76cafc7 [multi-context-plan-builder] First working example of async
execution
add 98487dee [multi-context-plan-builder] Fixed some sample operator tests
due an earlier tweak for making it serializable
add 578f2c47 [multi-context-plan-builder] Fixed some sample operator tests
due an earlier tweak for making it serializable vol.2
add c5d84eeb [multi-context-plan-builder] [wayang-benchmark] Removed
unnecessary planBuilder instances in parallel wayang job in
multicontext.apps.wordcount.Test
add 35a85635 Added missing apache licence headers and tweaks.
add b28bd45e [multi-context-plan-builder] Working on refactoring
AsyncDataQuanta
add dc074add [multi-context-plan-builder] Added additional APIs for the
asynchronous wayang jobs - still haven't decided on one
add 7e10e975 [multi-context-plan-builder] [asynchronous-jobs] Refactors
and documentation
add 33e337ba [multi-context-plan-builder] [asynchronous-jobs] Refactored
multicontext to use runAsync api. Moved main from MultiContextDataQuanta to
async. Various others refactors.
add 64e6e42b [multi-context-plan-builder] [asynchronous-jobs] Working on
finalizing refactors.
add 875f3a86 [multi-context-plan-builder] [asynchronous-jobs] Finalized
refactors. MultiContextDataQuanta api now runs on top of the async api.
add 425b4dd6 [multi-context-plan-builder] [asynchronous-jobs] Minor tweaks
in async benchmark mains
add d6fa9d8d [multi-context-plan-builder] OperatorSerializationTests:
Deactivated some logs
add 1be18f4b [multi-context-plan-builder] Added serialization support for
load estimators and reorganized mixins for serialization
add ebcfef2b [multi-context-plan-builder] Added serialization support for
cardinality estimators
add 35133b5c [multi-context-plan-builder] Parent process now passes its
own classpath. Async and multiContextPlanBuilder tests are now fixed.
add 724b9204 [multi-context-plan-builder] Tweaks
add c743bbc0 [multi-context-plan-builder] Added documentation (readme.md)
for async and multicontext operations. Also, tweaked the corresponding
benchmarks.
add 29c60770 [multi-context-plan-builder] [wayang-api-scala-java]
README.md tweaks.
add 0c9958ea Merged wayang 0.7.1
add b50656c3 Added readme on how to merge changes from the public wayang
repo
add 38563acf Added licenses
add f27eb6c0 [wayang-api-scala-java] Removed some logs in tests
add 49421909 Merged multicontext
add f4b4a5a5 Merged multicontext tweak
add 070b0d5e Merged multicontext cleanup
add 0d0fae91 Renamed BlossomContext to MultiContext
add b67b404e Merge pull request #429 from damik3/main
No new revisions were added by this update.
Summary of changes:
wayang-api/wayang-api-scala-java/README.md | 364 +++++++++++++++
wayang-api/wayang-api-scala-java/pom.xml | 21 +-
.../scala/org/apache/wayang/api/DataQuanta.scala | 6 +-
.../org/apache/wayang/api/DataQuantaBuilder.scala | 8 +-
.../scala/org/apache/wayang/api/MultiContext.scala | 73 +++
.../apache/wayang/api/MultiContextDataQuanta.scala | 179 +++++++
.../wayang/api/MultiContextPlanBuilder.scala | 187 ++++++++
.../scala/org/apache/wayang/api/PlanBuilder.scala | 4 +-
.../wayang/api/async/DataQuantaAsyncResult.scala | 31 +-
.../wayang/api/async/DataQuantaImplicits.scala | 99 ++++
.../scala/org/apache/wayang/api/async/Main.scala | 85 ++++
.../wayang/api/async/PlanBuilderImplicits.scala | 49 ++
.../org/apache/wayang/api/async/package.scala | 140 ++++++
.../main/scala/org/apache/wayang/api/package.scala | 12 +-
.../api/serialization/SerializationUtils.scala | 203 ++++++++
.../wayang/api/serialization/TempFileUtils.scala | 65 +++
.../GenericSerializableDeserializer.scala | 27 +-
.../GenericSerializableSerializer.scala | 28 +-
.../MultiContextDeserializer.scala | 98 ++++
.../customserializers/MultiContextSerializer.scala | 63 +++
.../customserializers/OperatorDeserializer.scala | 430 +++++++++++++++++
.../customserializers/PlatformDeserializer.scala | 47 ++
.../customserializers/PlatformSerializer.scala} | 18 +-
.../mixins/ConfigurationAndContextMixIns.scala | 122 +++++
.../api/serialization/mixins/DataTypeMixIns.scala | 68 +++
.../serialization/mixins/DescriptorMixIns.scala | 118 +++++
.../api/serialization/mixins/EstimatorMixIns.scala | 124 +++++
.../serialization/mixins/IgnoreLoggerMixIn.scala | 11 +-
.../api/serialization/mixins/OperatorMixIns.scala | 134 ++++++
.../api/serialization/mixins/ProviderMixIns.scala | 85 ++++
.../api/serialization/mixins/SlotMixIns.scala | 45 ++
.../serialization/OperatorSerializationTests.scala | 516 +++++++++++++++++++++
.../serialization/OtherSerializationTests.scala | 370 +++++++++++++++
.../api/serialization/SerializationTestBase.scala | 150 ++++++
.../org/apache/wayang/async/apps/WordCount.scala | 75 +++
.../apache/wayang/multicontext/apps/package.scala | 44 +-
.../wayang/multicontext/apps/tpch/Query1.scala | 131 ++++++
.../multicontext/apps/wordcount/WordCount.scala | 71 +++
.../apps/wordcount/WordCountCombineEach.scala | 69 +++
.../apps/wordcount/WordCountWithMerge.scala | 79 ++++
.../wordcount/WordCountWithTargetPlatforms.scala | 69 +++
.../wayang/basic/operators/SampleOperator.java | 17 +-
.../wayang/core/function/FunctionDescriptor.java | 31 +-
.../cardinality/DefaultCardinalityEstimator.java | 9 +-
.../core/optimizer/costs/DefaultLoadEstimator.java | 5 +-
.../wayang/core/optimizer/costs/LoadEstimator.java | 3 +-
.../optimizer/costs/LoadProfileEstimators.java | 10 +-
.../costs/NestableLoadProfileEstimator.java | 7 +-
.../DefaultCardinalityEstimatorTest.java | 9 +-
.../flink/operators/FlinkSampleOperator.java | 3 +-
.../java/operators/JavaRandomSampleOperator.java | 3 +-
.../operators/JavaReservoirSampleOperator.java | 3 +-
wayang-platforms/wayang-jdbc-template/pom.xml | 5 +
.../wayang/jdbc/operators/SqlToRddOperator.java | 5 +-
wayang-platforms/wayang-postgres/pom.xml | 5 +
.../operators/SparkBernoulliSampleOperator.java | 3 +-
.../SparkRandomPartitionSampleOperator.java | 3 +-
.../SparkShufflePartitionSampleOperator.java | 3 +-
58 files changed, 4485 insertions(+), 157 deletions(-)
create mode 100644 wayang-api/wayang-api-scala-java/README.md
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/MultiContext.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/MultiContextDataQuanta.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/MultiContextPlanBuilder.scala
copy
wayang-profiler/src/main/java/org/apache/wayang/profiler/log/sampling/Sampler.java
=>
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/async/DataQuantaAsyncResult.scala
(51%)
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/async/DataQuantaImplicits.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/async/Main.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/async/PlanBuilderImplicits.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/async/package.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/SerializationUtils.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/TempFileUtils.scala
copy
wayang-benchmark/src/main/scala/org/apache/wayang/apps/simwords/ScrubFunction.scala
=>
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/customserializers/GenericSerializableDeserializer.scala
(55%)
copy
wayang-benchmark/src/main/scala/org/apache/wayang/apps/tpch/CsvUtils.scala =>
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/customserializers/GenericSerializableSerializer.scala
(54%)
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/customserializers/MultiContextDeserializer.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/customserializers/MultiContextSerializer.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/customserializers/OperatorDeserializer.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/customserializers/PlatformDeserializer.scala
copy
wayang-api/{wayang-api-python/src/main/java/org/apache/wayang/api/python/function/PythonCode.java
=>
wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/customserializers/PlatformSerializer.scala}
(63%)
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/ConfigurationAndContextMixIns.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/DataTypeMixIns.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/DescriptorMixIns.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/EstimatorMixIns.scala
copy
wayang-commons/wayang-basic/src/main/java/org/apache/wayang/basic/model/LinearRegressionModel.java
=>
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/IgnoreLoggerMixIn.scala
(78%)
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/OperatorMixIns.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/ProviderMixIns.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/main/scala/org/apache/wayang/api/serialization/mixins/SlotMixIns.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/test/scala/org/apache/wayang/api/serialization/OperatorSerializationTests.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/test/scala/org/apache/wayang/api/serialization/OtherSerializationTests.scala
create mode 100644
wayang-api/wayang-api-scala-java/src/test/scala/org/apache/wayang/api/serialization/SerializationTestBase.scala
create mode 100644
wayang-benchmark/src/main/scala/org/apache/wayang/async/apps/WordCount.scala
copy
wayang-commons/wayang-basic/src/main/java/org/apache/wayang/basic/model/op/Mean.java
=>
wayang-benchmark/src/main/scala/org/apache/wayang/multicontext/apps/package.scala
(53%)
create mode 100644
wayang-benchmark/src/main/scala/org/apache/wayang/multicontext/apps/tpch/Query1.scala
create mode 100644
wayang-benchmark/src/main/scala/org/apache/wayang/multicontext/apps/wordcount/WordCount.scala
create mode 100644
wayang-benchmark/src/main/scala/org/apache/wayang/multicontext/apps/wordcount/WordCountCombineEach.scala
create mode 100644
wayang-benchmark/src/main/scala/org/apache/wayang/multicontext/apps/wordcount/WordCountWithMerge.scala
create mode 100644
wayang-benchmark/src/main/scala/org/apache/wayang/multicontext/apps/wordcount/WordCountWithTargetPlatforms.scala
