snuyanzin commented on code in PR #27776: URL: https://github.com/apache/flink/pull/27776#discussion_r2965493869
########## AGENTS.md: ########## @@ -0,0 +1,326 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Flink AI Agent Instructions + +This file provides guidance for AI coding agents working with the Apache Flink codebase. + +## Prerequisites + +- Java 11, 17 (default), or 21 +- Maven 3.8.6 (Maven wrapper `./mvnw` included; prefer it) +- Git +- Unix-like environment (Linux, macOS, WSL, Cygwin) + +## Commands + +### Build + +- Full build (Java 17 default): `./mvnw clean package -DskipTests -Djdk17 -Pjava17-target` +- Java 11: `./mvnw clean package -DskipTests -Djdk11 -Pjava11-target` +- Java 21: `./mvnw clean package -DskipTests -Djdk21 -Pjava21-target` +- Full build with tests: `./mvnw clean verify` +- Single module: `./mvnw clean package -DskipTests -pl flink-runtime` +- Single module with tests: `./mvnw clean verify -pl flink-runtime` + +### Testing + +- Single test class: `./mvnw -pl flink-runtime -Dtest=MyTestClass test` +- Single test method: `./mvnw -pl flink-runtime -Dtest=MyTestClass#myTestMethod test` + +### Code Quality + +- Format code (Java + Scala): `./mvnw spotless:apply` +- Check formatting: `./mvnw spotless:check` +- Checkstyle config: `tools/maven/checkstyle.xml` (v10.18.2) + +## Repository Structure + +Every module from the root pom.xml, organized by function. Flink provides three main user-facing APIs (recommended in this order: SQL, Table API, DataStream API) plus a newer DataStream v2 API. + +### Core Infrastructure + +- `flink-annotations` — Stability annotations (`@Public`, `@PublicEvolving`, `@Internal`, `@Experimental`) and `@VisibleForTesting` +- `flink-core-api` — Core API interfaces (functions, state, types) shared by all APIs +- `flink-core` — Core implementation (type system, serialization, memory management, configuration) +- `flink-runtime` — Distributed runtime (JobManager, TaskManager, scheduling, network, state) +- `flink-clients` — CLI and client-side job submission +- `flink-rpc/` — RPC framework + - `flink-rpc-core` — RPC interfaces + - `flink-rpc-akka`, `flink-rpc-akka-loader` — Pekko-based RPC implementation + +### SQL / Table API (recommended API for most users) + +- `flink-table/` + - `flink-sql-parser` — SQL parser (extends Calcite SQL parser) + - `flink-table-common` — Shared types, descriptors, catalog interfaces + - `flink-table-api-java` — Table API for Java + - `flink-table-api-scala` — Table API for Scala + - `flink-table-api-bridge-base`, `flink-table-api-java-bridge`, `flink-table-api-scala-bridge` — Bridges between Table and DataStream APIs + - `flink-table-api-java-uber` — Uber JAR for Table API + - `flink-table-planner` — SQL/Table query planning and optimization (Calcite-based) + - `flink-table-planner-loader`, `flink-table-planner-loader-bundle` — Classloader isolation for planner + - `flink-table-runtime` — Runtime operators for Table/SQL queries + - `flink-table-calcite-bridge` — Bridge to Apache Calcite + - `flink-sql-gateway-api`, `flink-sql-gateway` — SQL Gateway for remote SQL execution + - `flink-sql-client` — Interactive SQL CLI + - `flink-sql-jdbc-driver`, `flink-sql-jdbc-driver-bundle` — JDBC driver for SQL Gateway + - `flink-table-code-splitter` — Code generation utilities + - `flink-table-test-utils` — Test utilities for Table/SQL + +### DataStream API (original streaming API) + +- `flink-streaming-java` — DataStream API and stream processing operator implementations + +### DataStream API v2 (newer event-driven API) + +- `flink-datastream-api` — DataStream v2 API definitions +- `flink-datastream` — DataStream v2 API implementation + +### Connectors (in-tree) + +- `flink-connectors/` + - `flink-connector-base` — Base classes for source/sink connectors + - `flink-connector-files` — Unified file system source and sink + - `flink-connector-datagen` — DataGen source for testing + - `flink-connector-datagen-test` — Tests for DataGen connector + - `flink-hadoop-compatibility` — Hadoop InputFormat/OutputFormat compatibility + - `flink-file-sink-common` — Common file sink utilities +- Most connectors (Kafka, JDBC, Elasticsearch, etc.) live in separate repos under [github.com/apache](https://github.com/apache); see README.md for the full list + +### Formats + +- `flink-formats/` + - `flink-json`, `flink-csv`, `flink-avro`, `flink-parquet`, `flink-orc`, `flink-protobuf` — Serialization formats + - `flink-avro-confluent-registry` — Avro with Confluent Schema Registry + - `flink-sequence-file`, `flink-compress`, `flink-hadoop-bulk`, `flink-orc-nohive` — Hadoop-related formats + - `flink-format-common` — Shared format utilities + - `flink-sql-json`, `flink-sql-csv`, `flink-sql-avro`, `flink-sql-parquet`, `flink-sql-orc`, `flink-sql-protobuf` — SQL-layer format integrations + - `flink-sql-avro-confluent-registry` — SQL-layer Avro with Confluent Schema Registry + +### State Backends + +- `flink-state-backends/` + - `flink-statebackend-rocksdb` — RocksDB state backend + - `flink-statebackend-forst` — ForSt state backend (experimental, RocksDB-based with JNI bindings) + - `flink-statebackend-heap-spillable` — Heap-based spillable state backend + - `flink-statebackend-changelog` — Changelog state backend + - `flink-statebackend-common` — Shared state backend utilities +- `flink-dstl/flink-dstl-dfs` — State changelog storage (DFS-based persistent changelog for incremental checkpointing) + +### File Systems + +- `flink-filesystems/` + - `flink-hadoop-fs` — Hadoop FileSystem abstraction + - `flink-s3-fs-hadoop`, `flink-s3-fs-presto`, `flink-s3-fs-base` — S3 file systems + - `flink-oss-fs-hadoop` — Alibaba OSS + - `flink-azure-fs-hadoop` — Azure Blob Storage + - `flink-gs-fs-hadoop` — Google Cloud Storage + - `flink-fs-hadoop-shaded` — Shaded Hadoop dependencies + +### Queryable State + +- `flink-queryable-state/` + - `flink-queryable-state-runtime` — Server-side queryable state service + - `flink-queryable-state-client-java` — Client for querying operator state from running jobs + +### Deployment + +- `flink-kubernetes` — Kubernetes integration +- `flink-yarn` — YARN integration +- `flink-dist`, `flink-dist-scala` — Distribution packaging +- `flink-container` — Container entry-point and utilities for containerized deployments + +### Metrics + +- `flink-metrics/` + - `flink-metrics-core` — Metrics API and core implementation + - Reporter implementations: `flink-metrics-jmx`, `flink-metrics-prometheus`, `flink-metrics-datadog`, `flink-metrics-statsd`, `flink-metrics-graphite`, `flink-metrics-influxdb`, `flink-metrics-slf4j`, `flink-metrics-dropwizard`, `flink-metrics-otel` + +### Libraries + +- `flink-libraries/` + - `flink-cep` — Complex Event Processing + - `flink-state-processing-api` — Offline state access (savepoint reading/writing) + +### Other + +- `flink-models` — AI model integration (sub-modules: `flink-model-openai`, `flink-model-triton`) +- `flink-python` — PyFlink (Python API) +- `flink-runtime-web` — Web UI for JobManager dashboard +- `flink-external-resources` — External resource management (e.g., GPU) +- `flink-docs` — Documentation sources +- `flink-examples` — Example programs +- `flink-quickstart` — Maven archetype for new projects +- `flink-walkthroughs` — Tutorial walkthrough projects + +### Testing + +- `flink-tests` — Integration tests +- `flink-end-to-end-tests` — End-to-end tests +- `flink-test-utils-parent` — Test utility classes +- `flink-yarn-tests` — YARN-specific tests +- `flink-fs-tests` — FileSystem tests +- `flink-architecture-tests` — ArchUnit architectural boundary tests +- `tools/ci/flink-ci-tools` — CI tooling + +## Architecture Boundaries + +1. **Client** submits jobs to the cluster. Submission paths include the CLI (`bin/flink run` via `flink-clients`), the SQL Client (`bin/sql-client.sh` via `flink-sql-client`), the SQL Gateway (`flink-sql-gateway`, also accessible via JDBC driver), the REST API (direct HTTP to JobManager), programmatic execution (`StreamExecutionEnvironment.execute()` or `TableEnvironment.executeSql()`), and PyFlink (`flink-python`, wraps the Java APIs). +2. **JobManager** (`flink-runtime`) orchestrates execution: receives jobs, creates the execution graph, manages scheduling, coordinates checkpoints, and handles failover. Never runs user code directly. +3. **TaskManager** (`flink-runtime`) executes the user's operators in task slots. Manages network buffers, state backends, and I/O. +4. **Table Planner** (`flink-table-planner`) translates SQL/Table API programs into DataStream programs. The planner is loaded in a separate classloader (`flink-table-planner-loader`) to isolate Calcite dependencies. +5. **Connectors** communicate with external systems. Source connectors implement the `Source` API (FLIP-27); sinks implement the `SinkV2` API. Most connectors are externalized to separate repositories. +6. **State Backends** persist keyed state and operator state. RocksDB is the primary backend for production use. +7. **Checkpointing** provides exactly-once guarantees. The JobManager coordinates barriers through the data stream; TaskManagers snapshot local state to a distributed file system. + +Key separations: + +- **Planner vs Runtime:** The table planner generates code and execution plans; the runtime executes them. Changes to planning logic live in `flink-table-planner`; changes to runtime operators live in `flink-table-runtime` or `flink-streaming-java`. +- **API vs Implementation:** Public API surfaces (`flink-core-api`, `flink-datastream-api`, `flink-table-api-java`) are separate from implementation modules. API stability annotations control what users can depend on. +- **ArchUnit enforcement:** `flink-architecture-tests/` contains ArchUnit tests that enforce module boundaries. New violations should be avoided; if unavoidable, follow the freeze procedure in `flink-architecture-tests/README.md`. + +## Common Change Patterns + +This section maps common types of Flink changes to the modules they touch and the verification they require. + +### Adding a new SQL built-in function + +1. Register in `flink-table-common` in `BuiltInFunctionDefinitions.java` (definition, input/output type strategies, runtime class reference) +2. Implement in `flink-table-runtime` under `functions/scalar/` (extend `BuiltInScalarFunction`, implement `eval()` method) +3. Add tests in `flink-table-planner` (planner integration) and `flink-table-runtime` (unit tests) +4. Document in `flink-docs` +5. Verify: planner tests + runtime tests + SQL ITCase + +### Adding a new configuration option + +1. Define `ConfigOption<T>` in the relevant config class (e.g., `ExecutionConfigOptions.java` in `flink-table-api-java`) +2. Use `ConfigOptions.key("table.exec....")` builder with type, default value, and description +3. Add `@Documentation.TableOption` annotation for auto-generated docs +4. Document in `flink-docs` if user-facing +5. Verify: unit test for default value, ITCase for behavior change + +### Adding a new table operator (e.g., join type, aggregate) + +1. **Planner rule** in `flink-table-planner` under `plan/rules/physical/stream/` (match conditions, transform to physical node) Review Comment: do not write new tests in scala -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
