Copilot commented on code in PR #802: URL: https://github.com/apache/wayang/pull/802#discussion_r3919034015
########## wayang-docs/src/main/resources/using_wayang/scalable_deep_learning.md: ########## @@ -0,0 +1,111 @@ +--- +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +layout: default +title: "Scalable Deep Learning" +previous: + url: /using_wayang/cost_model_calibration/ + title: Cost Model Calibration +next: + url: /how_contribute/ + title: How To Contribute +menu: + using: + weight: 7 +--- + +# Scalable Deep Learning in Apache Wayang + +Apache Wayang provides first-class support for scalable deep learning, bridging big data processing platforms with deep learning frameworks. + +Through the `wayang-tensorflow` platform module and core deep learning abstractions, Wayang enables distributed data preprocessing (e.g., via Spark or Java Streams) combined seamlessly with GPU/CPU accelerated neural network training and batch inference. + +--- + +## Core Abstractions + +### 1. `DLModel` +`org.apache.wayang.basic.model.DLModel` represents a deep neural network model. It encapsulates the computational graph, layer definitions (e.g., Dense/Linear, Conv2D, Conv3D, BatchNorm, ConvLSTM), and trainable weights. + +### 2. `DLTrainingOperator` +`DLTrainingOperator` trains a deep neural network model on an input dataset. It takes training features and labels as input streams or tensors, executes training epochs with specified loss functions and optimizers, and produces an updated `DLModel`. + +### 3. `PredictOperator` +`PredictOperator` performs high-throughput batch inference. It applies an existing `DLModel` to incoming input data tensors and produces predicted outputs or probability distributions. + +--- + +## The TensorFlow Platform (`wayang-tensorflow`) + +Wayang includes a dedicated platform adapter for TensorFlow (`wayang-tensorflow`), utilizing Java bindings and native acceleration to execute deep learning operators on CPUs and GPUs. + +### Maven Dependency +To use TensorFlow capabilities in your Wayang application, add the following dependency: Review Comment: This instruction doesn't mention that `-SNAPSHOT` versions require configuring an additional repository; consider calling out the `WAYANG_VERSION` placeholder and snapshot repository requirement to prevent copy/paste build failures. This issue also appears on line 62 of the same file. ########## wayang-docs/src/main/resources/getting_start/how_build/index.md: ########## @@ -17,11 +17,63 @@ license: | layout: default title: "How to Build Wayang" previous: - url: / - title: previous + url: /getting_start/ + title: Getting Started next: - url: / - title: next + url: /getting_start/how_build/build_step/ + title: Step by Step Building Wayang +menu: + getting_start: + weight: 0 +--- + +# How to Build Apache Wayang + +This guide details the system prerequisites, environment setup, and Maven commands required to build Apache Wayang from source. + +--- + +## Requirements + +Before building Apache Wayang from source, ensure your environment meets the following specifications: + +- **Java Development Kit (JDK)**: JDK 11 or JDK 17 (recommended: Eclipse Temurin OpenJDK 17 or OpenJDK 11). +- **Scala**: Version 2.12.x. +- **Apache Maven**: Maven 3.8.0 or newer (or use the included Maven wrapper `./mvnw` / `mvnw.cmd`). +- **Platform Prerequisites**: + - **Linux / macOS**: Standard development toolchains (`tar`, `gzip`). + - **Windows**: Requires Hadoop winutils binaries located in `%HADOOP_HOME%\bin\winutils.exe` if running Hadoop/Spark integration locally. + --- -How to Build Wayang +## Building from Source + +### Quick Build (Skipping Tests) +To build all Wayang modules and compile JARs without running test suites: + +```shell +$ git clone https://github.com/apache/wayang.git +$ cd wayang +$ ./mvnw clean install -DskipTests +``` + +### Full Build with Tests +To run unit and platform integration tests: + +```shell +$ ./mvnw clean install +``` + +--- + +## Build Profiles + +Wayang provides specialized Maven build profiles for assembling distributions and targeting execution environments: + +| Profile | Command | Purpose | +|---|---|---| +| `distro` | `./mvnw clean install -Pdistro` | Assembles the complete binary release archive in `wayang-assembly`. | +| `standalone` | `./mvnw clean install -Pstandalone` | Packages bundled dependencies so standalone applications do not need external cluster libraries. | +| `web-documentation` | `./mvnw site -pl wayang-docs -Pweb-documentation` | Builds the Jekyll documentation site. | + +For step-by-step guidance, see [Step by Step Building Wayang](build_step.md). Review Comment: Linking to the source filename (`build_step.md`) is inconsistent with the rest of the docs (which use pretty URLs like `how_build/`) and may break depending on Jekyll permalink settings; prefer linking to the generated page path. ########## wayang-docs/src/main/resources/getting_start/writting_wayang_plan/index.md: ########## @@ -8,20 +8,127 @@ license: | the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - + Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. layout: default -title: "Writting a Wayang Plan" +title: "Wayang Abstractions & Plans" previous: - url: / - title: previous + url: /getting_start/how_run/ + title: How to Run Wayang next: - url: / - title: next + url: /using_wayang/ + title: Using Wayang +menu: + getting_start: + weight: 3 +--- + +# Wayang Abstractions & Writing Plans + +Apache Wayang represents data processing applications as directed acyclic (or cyclic) dataflow graphs composed of platform-independent logical operators. At optimization time, Wayang translates these high-level operators into concrete execution operators mapped to the optimal underlying processing platforms (such as Java Streams, Apache Spark, Flink, or relational databases). + +--- + +## Core Operator Abstractions + +Wayang categorizes all data transformations into five fundamental operator archetypes: + +### 1. Source Operators +Source operators serve as the root entry points of a Wayang plan. They ingest raw data from external storage systems or collections without accepting input channels: +- **`TextFileSource`**: Reads text files from local disk, HDFS, or S3 line by line. +- **`TableSource`**: Queries structured relational database tables or views (e.g., PostgreSQL, SQLite). +- **`CollectionSource`**: Wraps in-memory Java/Scala collections into a distributed dataflow. + +### 2. Unary Operators +Unary operators process a single input dataset to produce a transformed output dataset: +- **`MapOperator`**: Applies a transformation function to each element (1-to-1). +- **`FilterOperator`**: Retains elements satisfying a boolean predicate. +- **`FlatMapOperator`**: Transforms each element into zero, one, or more output elements. +- **`ReduceByOperator`**: Aggregates elements sharing the same key. +- **`SortOperator`**: Orders records by specific sort keys. +- **`CountOperator`**: Calculates dataset cardinality. + +### 3. Binary Operators +Binary operators accept two distinct input datasets and produce an output dataset: +- **`JoinOperator`**: Performs relational inner or outer joins matching key extractors across two datasets. +- **`UnionAllOperator`**: Combines two datasets of identical data types into one unified stream. +- **`CartesianOperator`**: Computes the full cross-product of two input collections. +- **`IntersectOperator`**: Returns the set intersection between two collections. + +### 4. Loop Operators +Loop operators support iterative and recursive processing cycles, enabling complex graph and machine learning workflows: +- **`LoopOperator` / `DoWhileOperator`**: Iteratively applies a loop body until a convergence predicate or maximum iteration count is reached (crucial for algorithms like PageRank, K-Means, and Gradient Descent). +- **`RepeatOperator`**: Repeats a sub-plan for a fixed number of iterations. + +### 5. Sink Operators +Sink operators terminate the execution plan by writing output datasets to destination sinks or returning them to the driver application: +- **`TextFileSink`**: Serializes records to text files on local storage or distributed filesystems. +- **`CollectionSink` / `LocalCallbackSink`**: Collects records back into JVM memory or invokes a user callback for each produced element. + --- -Writting a Wayang Plan +## Assembling Plans with `PlanBuilder` + +The primary way to author Wayang applications in Java and Scala is through the fluent `PlanBuilder` API (`JavaPlanBuilder`). + +### Key PlanBuilder Components +- **`WayangContext`**: Holds execution configuration and registers the available platform plugins (e.g., `Java.basicPlugin()`, `Spark.basicPlugin()`). +- **`JavaPlanBuilder`**: Provides fluent factory methods to chain operators together. +- **`DataQuantaBuilder`**: Represents a intermediate dataset within the plan, providing transformation methods (`map`, `filter`, `reduceByKey`, `join`). Review Comment: The grammar is incorrect here: "a intermediate" should be "an intermediate". ########## wayang-docs/src/main/resources/index.md: ########## @@ -53,120 +53,44 @@ Important note: depending on the scala version the list of the supported platfor ## How to use Wayang -**Requirements.** -Apache Wayang (incubating) is built upon the foundations of Java 11 and Scala 2.12, providing a robust and versatile platform for data processing applications. If you intend to build Wayang from source, you will also need to have Apache Maven, the popular build automation tool, installed on your system. Additionally, be mindful that some of the processing platforms supported by Wayang may have their own specific installation requirements. +### Quick Navigation +- **[How to Build Wayang (Requirements)](getting_start/how_build/)**: System requirements (Java 11/17, Scala 2.12, Maven) and source build instructions. +- **[Wayang Abstractions & Plans](getting_start/writting_wayang_plan/)**: Learn about Source, Unary, Binary, Loop, and Sink operators and how to build plans using `JavaPlanBuilder`. +- **[Configuring Wayang](using_wayang/configuring_wayang/)**: System properties, platform parameters, and runtime configuration. +- **[Cost Model Calibration](using_wayang/cost_model_calibration/)**: Tuning the cost-based optimizer and calibrating load profile estimators. +- **[Scalable Deep Learning](using_wayang/scalable_deep_learning/)**: Deep learning with `DLModel` and the TensorFlow platform adapter. +- **[API JavaDocs](https://wayang.apache.org/docs/api/javadocs/)**: Official API reference documentation. -**Get Wayang.** -Wayang is available via Maven Central. To use it with Maven, for instance, include the following into you POM file: +--- + +### Get Wayang +Wayang is available via Maven Central. To use it with Maven, include the following into your `pom.xml`: Review Comment: This text says Wayang is available via Maven Central but the snippet uses a `-SNAPSHOT` version; snapshots are typically not published to Maven Central, so the instructions are misleading without mentioning the snapshot repository. This issue also appears on line 69 of the same file. ########## wayang-docs/src/main/resources/using_wayang/configuring_wayang.md: ########## @@ -0,0 +1,116 @@ +--- +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +layout: default +title: "Configuring Wayang" +previous: + url: /using_wayang/ + title: Using Wayang +next: + url: /using_wayang/cost_model_calibration/ + title: Cost Model Calibration +menu: + using: + weight: 5 +--- + +# Configuring Apache Wayang + +To enable Apache Wayang's smooth operation and intelligent optimization, you need to provide details about your processing platforms' capabilities, resources, and connection properties. + +While a default configuration is loaded automatically for local experimentation, creating a custom configuration properties file is recommended for fine-tuning performance or connecting to distributed execution engines. + +--- + +## Loading Custom Configurations + +You can load a custom configuration file into your application via the command-line JVM system property: + +```shell +$ java -Dwayang.configuration=file:///path/to/my/wayang.properties -cp ... my.app.Main +``` + +Alternatively, you can load or modify configurations programmatically in your Java or Scala application: + +```java +import org.apache.wayang.core.api.Configuration; +import org.apache.wayang.core.api.WayangContext; + +Configuration config = new Configuration("file:///path/to/my/wayang.properties"); +config.setProperty("wayang.spark.master", "spark://my-cluster:7077"); + +WayangContext wayangContext = new WayangContext(config); +``` + +--- + +## Key Configuration Properties + +### General Core Settings +| Property | Default | Description | +|---|---|---| +| `wayang.core.log.enabled` | `false` | Whether to log execution statistics to allow learning better cardinality and cost estimators for the optimizer. | +| `wayang.core.log.executions` | `~/.wayang/executions.json` | Destination path where execution times of operator groups are recorded. | +| `wayang.core.log.cardinalities` | `~/.wayang/cardinalities.json` | Destination path where cardinality measurements are stored. | +| `wayang.core.optimizer.instrumentation` | `OutboundInstrumentationStrategy` | Strategy for measuring intermediate cardinalities (`NoInstrumentationStrategy`, `OutboundInstrumentationStrategy`, or `FullInstrumentationStrategy`). | +| `wayang.core.optimizer.reoptimize` | `false` | Whether to progressively re-optimize execution plans at runtime based on actual intermediate cardinalities. | +| `wayang.basic.tempdir` | `file:///tmp` | Location used for storing temporary files, especially for inter-platform data exchanges. | + +--- + +### Java Streams Platform +| Property | Default | Description | +|---|---|---| +| `wayang.java.cpu.mhz` | `2700` | Clock frequency (MHz) of the processor executing the local JVM. | +| `wayang.java.hdfs.ms-per-mb` | `2.7` | Average throughput from HDFS to the local JVM in milliseconds per megabyte. | + +--- + +### Apache Spark Platform +| Property | Default | Description | +|---|---|---| +| `spark.master` | `local` | Spark master URL (e.g., `local[*]`, `spark://host:port`, or `yarn`). | +| `spark.app.name` | `Wayang App` | Spark application name. | +| `wayang.spark.cpu.mhz` | `2700` | CPU clock frequency (MHz) of the Spark worker nodes. | +| `wayang.spark.hdfs.ms-per-mb` | `2.7` | Throughput from HDFS to Spark workers (ms/MB). | +| `wayang.spark.network.ms-per-mb` | `8.6` | Average network throughput between Spark workers (ms/MB). | +| `wayang.spark.init.ms` | `4500` | Overhead time (ms) required for Spark context initialization. | + +--- + +### Relational Database Platforms (JDBC) + +#### PostgreSQL +| Property | Description | +|---|---| +| `wayang.postgres.jdbc.url` | JDBC connection URL (e.g., `jdbc:postgresql://localhost:5432/mydb`). | +| `wayang.postgres.jdbc.user` | Database user account name. | +| `wayang.postgres.jdbc.password` | Database password. | +| `wayang.postgres.cpu.mhz` | Clock frequency (MHz) of the PostgreSQL database server. | +| `wayang.postgres.cpu.cores` | Number of CPU cores available on the PostgreSQL database server. | + +#### SQLite3 +| Property | Description | +|---|---| +| `wayang.sqlite3.jdbc.url` | JDBC connection URL (e.g., `jdbc:sqlite:/path/to/database.db`). | +| `wayang.sqlite3.cpu.mhz` | Clock frequency (MHz) of the processor running SQLite. | +| `wayang.sqlite3.cpu.cores` | Available CPU cores on the SQLite host machine. | + +--- + +### Next Steps + +For advanced cost-based optimization and calibrating load profile estimator templates with historical workload metrics, see [Cost Model Calibration](cost_model_calibration.md). Review Comment: This link targets the Markdown source filename (`cost_model_calibration.md`) rather than the rendered page path; use the pretty URL form to avoid broken links under typical Jekyll permalink settings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
