GitHub user tuhaihe created a discussion: [Proposal] Remove `gpcontrib/gpmapreduce` starting from Cloudberry 3.0
### Proposers Dianjin Wang (@tuhaihe) ### Proposal Status Under Discussion ### Abstract This proposal suggests removing `gpmapreduce` from Apache Cloudberry, starting with the 3.0 development line. Concretely, on `main` (currently `3.0.0-devel`) we would delete: - the `gpcontrib/gpmapreduce/` directory (44 files, ~13,500 lines), - the `gpmapreduce` client binary that is currently installed into `$GPHOME/bin/`, - the `--enable-mapreduce` configure option and the unused `USE_MAPREDUCE` define, - the `gpmapreduce_help` file shipped by `gpMgmt/doc/`. The 2.x line (`REL_2_STABLE`, currently `2.2.0-incubating`) would **not** change. `gpmapreduce` keeps building and shipping there for the entire supported life of 2.x, so users who still depend on it have a supported place to stay. The removal takes effect in 3.0 and remains in effect for 3.x. ### Motivation ### 1. This functionality is no longer needed according to the commit `4d926ede502` Commit `4d926ede502` (2024-09-19) disabled the gpmapreduce regression test in `gpcontrib/Makefile`, with the following commit message: > As gpmapreduce is implemented for compatibility of execution of mapreduce, > which is no longer need now, we disable gpmapreduce regress test in this > commit. However, only the *test* was turned off. The *build* was not. Our release build script still passes `--enable-mapreduce` (`devops/build/automation/cloudberry/scripts/configure-cloudberry.sh`), and `gpcontrib/gpmapreduce/Makefile` still installs the binary into `$GPHOME/bin/gpmapreduce`. ### 2. Upstream removed it three years ago Greenplum deleted the same code in `6bc9129c90b` (2023-07-13): > It's decided that this functionality will not be provided for GPDB7 and > forward. Hence, deleting the code for the same. ### 3. The code is effectively frozen `git log gpcontrib/gpmapreduce` shows that every commit since the initial Cloudberry code dump has been mechanical: rebranding, typo fixes, Python 3 migration, and PostgreSQL merge fallout. The last functional change dates back to the Greenplum era. ### 4. Users have better alternatives today The workloads gpmapreduce was designed for in 2008 are well covered by plain SQL plus UDFs (PL/pgSQL, PL/Python, PL/Perl, PL/Container), external tables / gpfdist, and FDWs. We are not aware of active users of the YAML job-spec interface, and this proposal is partly a call for such users to speak up. ### Implementation The change is confined to build glue and one self-contained directory. Proposed as a single PR with atomic commits: 1. **Stop building and shipping `gpmapreduce`** — drop `--enable-mapreduce` from `devops/build/automation/cloudberry/scripts/configure-cloudberry.sh` (and update the feature comment at the top of that file), and remove the `mapreduce` target plus the `install` / `clean` / `distclean` branches from `gpcontrib/Makefile`. 2. **Delete `gpcontrib/gpmapreduce/`** — 44 files, ~13,500 lines. 3. **Remove the configure surface** — `--enable-mapreduce`, `USE_MAPREDUCE`, the `libyaml`/`yaml.h` checks that are specific to mapreduce, and the mapreduce-specific Perl requirement, in `configure.ac`; regenerate `configure`; drop `#undef USE_MAPREDUCE` from `src/include/pg_config.h.in`. 4. **Remove the help file** — `gpMgmt/doc/gpmapreduce_help` and its entry in `gpMgmt/doc/Makefile`. 5. **Drop the RAT exclusion** — remove `<exclude>gpcontrib/gpmapreduce/**</exclude>` from `pom.xml`. ### Rollout/Adoption Plan ### Version plan | Line | Behaviour | | --- | --- | | **2.x** (`REL_2_STABLE`, `2.2.0-incubating` and later 2.x releases) | **No change.** `gpmapreduce` continues to be built and shipped. Users who still depend on it stay on 2.x for the supported life of that line. | | **3.0** (`main`, `3.0.0-devel`) | Removed. | | **3.x** | Stays removed; not re-introduced. | This gives users an explicit, supported landing spot rather than an abrupt removal, while keeping the 3.x tree clean from the start of the line. ### Open questions for the community 1. Is anyone still running `gpmapreduce` in production? If so, on which Cloudberry version, and what is the migration blocker? ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! GitHub link: https://github.com/apache/cloudberry/discussions/2030 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
