kinghao007 opened a new issue, #5324: URL: https://github.com/apache/linkis/issues/5324
### Search before asking - [x] I searched the [issues](https://github.com/apache/linkis/issues) and found no similar issues. ### Linkis Component linkis-engineconn-plugins ### What happened **English:** Linkis currently lacks native support for Apache Kylin, the leading massive-scale OLAP engine designed for trillion-row data analysis. Kylin is widely adopted in financial and telecom industries (ICBC, CCB, CMB, China Mobile, China Telecom) for regulatory reports, customer profiling, and risk analysis with PB-scale data. **Market Demand:** - **Massive-Scale Support**: Specifically designed for trillion-row data analysis with sub-second query response through pre-computation Cube technology - **Financial Industry Standard**: Core banks (ICBC, CCB, CMB) rely on Kylin for regulatory reports (EAST, T+1 reports) with tens of TB data - **Telecom Industry Adoption**: China Mobile and China Telecom use Kylin for user behavior analysis with 10TB-100TB+ data volumes - **Apache Top-Level Project**: Mature ecosystem with commercial support from Kyligence - **Pre-computation Advantage**: Converts complex queries into simple KV lookups for guaranteed performance **Strategic Value:** Kylin complements existing engines by addressing fixed-dimension massive-scale analysis: - **Kylin**: PB-scale, sub-second, fixed dimensions → regulatory reports, massive-scale dashboards - **Doris**: TB-PB scale, second-level, flexible → real-time multi-dimensional analysis - **ClickHouse**: TB-scale, sub-second, flexible → real-time wide-table queries - **Presto**: PB-scale, minute-level, extremely flexible → ad-hoc queries --- **中文:** Linkis目前缺乏对Apache Kylin的原生支持，Kylin是领先的超大规模OLAP引擎，专为万亿级数据分析设计。Kylin在金融和电信行业（工商银行、建设银行、招商银行、中国移动、中国电信）被广泛采用，用于监管报表、客户画像和PB级数据的风险分析。 **市场需求:** - **超大规模支持**: 专为万亿级数据分析设计，通过预计算Cube技术实现亚秒级查询响应 - **金融行业标配**: 核心银行（工商银行、建设银行、招商银行）依赖Kylin处理数十TB的监管报表（EAST、T+1报表） - **电信行业采用**: 中国移动和中国电信使用Kylin进行10TB-100TB+数据量的用户行为分析 - **Apache顶级项目**: 成熟的生态系统，Kyligence提供商业支持 - **预计算优势**: 将复杂查询转化为简单的KV查找，保证性能 **战略价值:** Kylin通过解决固定维度超大规模分析来补充现有引擎： - **Kylin**: PB级、亚秒级、固定维度 → 监管报表、超大规模仪表板 - **Doris**: TB-PB级、秒级、灵活 → 实时多维分析 - **ClickHouse**: TB级、亚秒级、灵活 → 实时大宽表查询 - **Presto**: PB级、分钟级、极灵活 → 即席查询 ### What you expected to happen **English:** Linkis should provide an Apache Kylin engine plugin with the following capabilities: 1. **Query Support:** - SQL query interface for Kylin Cubes - Support for Kylin 4.x and Kylin 5.x versions - MDX query support (optional) - Query routing to pre-built Cubes 2. **Cube Management:** - Trigger Cube build jobs from Linkis - Monitor Cube build progress - Cube refresh and incremental build support - Cube metadata query and management 3. **Data Operations:** - Query pre-built Cubes with standard SQL - Support for complex aggregations - Drill-down and roll-up operations - Time-based filtering and analysis 4. **Integration with Linkis:** - Unified task submission interface - Resource management for Cube builds - Permission control integration - Metadata catalog integration 5. **Performance Features:** - Query result caching - Connection pooling - Cube query optimization - Pushdown computation to Kylin --- **中文:** Linkis应该提供Apache Kylin引擎插件，具备以下能力： 1. **查询支持:** - Kylin Cube的SQL查询接口 - 支持Kylin 4.x和Kylin 5.x版本 - MDX查询支持（可选） - 查询路由到预构建的Cube 2. **Cube管理:** - 从Linkis触发Cube构建作业 - 监控Cube构建进度 - Cube刷新和增量构建支持 - Cube元数据查询和管理 3. **数据操作:** - 使用标准SQL查询预构建的Cube - 支持复杂聚合 - 下钻和上卷操作 - 基于时间的过滤和分析 4. **与Linkis集成:** - 统一的任务提交接口 - Cube构建的资源管理 - 权限控制集成 - 元数据目录集成 5. **性能特性:** - 查询结果缓存 - 连接池 - Cube查询优化 - 下推计算到Kylin ### How to reproduce **English:** Current situation: 1. Users need to manually access Kylin through REST API or web UI 2. No unified interface for Cube management within Linkis 3. Cannot leverage Linkis's resource management for Cube builds 4. Isolated from Linkis's permission and metadata systems Use case example: ```sql -- Financial regulatory report: Daily transaction summary by region -- Data volume: 50TB+, Cube pre-built for fixed dimensions SELECT report_date, region_code, branch_code, SUM(transaction_amount) as total_amount, COUNT(transaction_id) as transaction_count, COUNT(DISTINCT customer_id) as customer_count FROM transaction_cube WHERE report_date >= '2024-01-01' AND report_date <= '2024-12-31' AND region_code IN ('BJ', 'SH', 'GZ', 'SZ') GROUP BY report_date, region_code, branch_code ORDER BY report_date DESC, total_amount DESC; -- Cube build job (currently cannot be submitted through Linkis) -- BUILD CUBE transaction_cube START('2024-12-01') END('2024-12-31') -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
