aiceflower opened a new pull request, #5345:
URL: https://github.com/apache/linkis/pull/5345
## What is the purpose of the change
This PR merges WeDataSphere's production-tested enhancements and bug fixes
into the Apache Linkis mainline. These changes have been extensively tested and
used in production environments at Bank of China.
The merge includes improvements across multiple modules:
- **EngineConn**: Enhanced executor management, UDF loading, and Python
module support
- **Entrance**: Improved job scheduling with FIFO and parallel consumer
managers
- **Manager**: Better resource management, label handling, and engine plugin
loading
- **Storage**: Multi-cloud storage support (HDFS, S3, Azure Blob, OSS, Local)
- **Client**: UJES client improvements and CLI enhancements
- **Common Utilities**: Security utilities (AES, SHA, RSA), parameter utils,
and variable operations
- **Build System**: Optimized build scripts with parallel compilation support
## Related issues/PRs
Related issues: N/A (code merge)
## Brief change log
### Core Modules
- **linkis-common**: Added security utilities (AES, SHA, RSA), parameter
utils, variable type system
- **linkis-storage**: Enhanced multi-cloud storage support with Azure Blob,
S3, OSS filesystems
- **linkis-rpc**: Improved RPC consumer transformer and serialization
- **linkis-scheduler**: Added FIFO and parallel consumer managers for better
resource utilization
- **linkis-protocol**: Added task constants and immutable pair utility
### Computation Governance
- **EngineConn**:
- Added Python module loading hook
- Added UDF loading hook
- Improved executor context management
- Enhanced timing monitor service
- **Entrance**:
- Added WebSocket service for real-time updates
- Improved request interceptors
- Enhanced SQL explanation and comment handling
- Added parallel orchestrator plugin
- **Manager**:
- Improved engine plugin loader
- Enhanced label management system
- Better resource persistence handling
- **Client**:
- UJES client improvements
- CLI command template enhancements
### Infrastructure
- **Quick Build Script**: Added optimized build script with 40-50%
performance improvement
- Parallel compilation with configurable threads
- Multi-version support (Spark 2.4/3.x, Hadoop 2.7/3.3)
- Three-stage hybrid build strategy
- **Documentation**: Added AI prompt guidelines, error codes, and
configuration docs
- **CI/CD**: Enhanced GitHub Actions workflows for better build reliability
### Bug Fixes and Optimizations
- Fixed Hive connection currency issues
- Improved kerberos authentication handling
- Fixed various compilation errors and dependency conflicts
- Optimized gateway and MyBatis configurations
- Enhanced logging and monitoring capabilities
## Build Configuration
| Profile | Spark | Scala | Hadoop | Hive |
|---------|-------|-------|--------|------|
| Default | 2.4.3 | 2.11.12 | 2.7.2 | 2.3.3 |
| `-Pspark-3` | 3.4.4 | 2.12.17 | 2.7.2 | 2.3.3 |
| `-Papache` | 3.2.1 | 2.12.17 | 3.3.4 | 3.1.3 |
## Checklist
- [x] I have read the [Contributing Guidelines on pull
requests](https://github.com/apache/linkis/blob/dev-1.9.0/CONTRIBUTING.md)
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [ ] I have added tests corresponding to this change (existing tests cover
the changes)
- [ ] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible
## Verification
These changes have been:
- Tested in WeDataSphere's production environment
- Verified with multiple Spark and Hadoop version combinations
- Validated through CI/CD pipelines
To build with different profiles:
```bash
# Default build (Spark 2.4.3)
./quick-build.sh
# Spark 3 build
./quick-build.sh -Pspark-3
# Apache release build
./quick-build.sh --v3
```
## Statistics
- **Files Changed**: 475
- **Insertions**: ~22,100 lines
- **Deletions**: ~7,870 lines
- **Modules Affected**: linkis-commons, linkis-computation-governance,
linkis-dist, docs
---
**Note**: This is a comprehensive merge of production-tested code from
WeDataSphere's internal fork, bringing significant improvements in stability,
performance, and feature completeness to Apache Linkis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
