This is an automated email from the ASF dual-hosted git repository. aiceflower pushed a commit to branch release-0.9.4 in repository https://gitbox.apache.org/repos/asf/linkis.git
commit 76771f4b8f95b1384393713cd471e57aa6fb7399 Author: David hua <[email protected]> AuthorDate: Wed Jun 3 21:47:32 2020 +0800 Branch 0.9.4[DataSource] (#426) * #299 Provide configuration service in module * Shell script is truncated when executing sqoop command close #315 * Fix bug of bml server log config which writes an absolute log path (#319) * Contributing guide in Chinese. * Fix jdbc engine hanged after executing multiple error SQL (#336) close #334 * fix configuration issue of StorageConfiguration. * 1.if job is successful, errCode and errDesc need to be null * Remove error stack information * fix CRLF bug * limit getDirFileTrees interface * update path check * add trigger for path check * #406 Put the code in module "datasource manager" together Co-authored-by: chaogefeng <[email protected]> Co-authored-by: cooperyang <[email protected]> Co-authored-by: sargentti <[email protected]> Co-authored-by: linfeng <[email protected]> Co-authored-by: tangjianfeng <[email protected]> --- Contributing_CN.md | 172 +++++++++++++++++ bml/bmlserver/conf/log4j2.xml | 4 +- .../datasourcemanager/server/bin/start-dsm.sh | 38 ++++ .../datasourcemanager/server/bin/stop-dsm.sh | 47 +++++ .../datasourcemanager/server/conf}/application.yml | 0 .../server/conf}/linkis.properties | 0 .../server/conf}/log4j.properties | 0 .../server/conf}/log4j2.xml | 0 .../datasourcemanager}/server/pom.xml | 41 +++- .../server/src/main/assembly/distribution.xml | 206 ++++++++++++++++++++ .../CustomMultiPartFormDataTransformer.java | 0 .../core/formdata/FormDataTransformerFactory.java | 0 .../core/formdata/FormStreamContent.java | 0 .../formdata/MultiPartFormDataTransformer.java | 0 .../core/service/BmlAppService.java | 0 .../core/service/DataSourceInfoService.java | 0 .../core/service/DataSourceOpService.java | 0 .../core/service/DataSourceRelateService.java | 0 .../core/service/MetadataOperateService.java | 0 .../core/service/impl/BmlAppServiceImpl.java | 0 .../service/impl/DataSourceInfoServiceImpl.java | 0 .../service/impl/DataSourceRelateServiceImpl.java | 0 .../service/impl/MetadataOperateServiceImpl.java | 0 .../validate/DataSourceParameterValidator.java | 0 .../core/validate/ParameterValidateException.java | 0 .../core/validate/ParameterValidateStrategy.java | 0 .../core/validate/ParameterValidator.java | 0 .../strategy/RegExpParameterValidateStrategy.java | 0 .../strategy/TypeParameterValidateStrategy.java | 0 .../datasourcemanager/core/vo/DataSourceEnvVo.java | 0 .../datasourcemanager/core/vo/DataSourceVo.java | 0 .../datasourcemanager/core/vo/PageViewVo.java | 0 .../core/receivers/DsmReceiver.scala | 0 datasource/metadatamanager/elasticsearch/pom.xml | 78 -------- .../metadatamanager/service/ElasticConnection.java | 140 -------------- .../service/ElasticMetaService.java | 87 --------- .../service/ElasticParamsMapper.java | 32 ---- .../src/main/resources/application.yml | 23 --- .../src/main/resources/linkis.properties | 21 --- .../service/receiver/ElasticReceiver.scala | 29 --- datasource/metadatamanager/mysql/service/pom.xml | 83 -------- .../metadatamanager/service/SqlConnection.java | 210 --------------------- .../metadatamanager/service/SqlMetaService.java | 79 -------- .../metadatamanager/service/SqlParamsMapper.java | 38 ---- .../service/src/main/resources/application.yml | 23 --- .../service/src/main/resources/linkis.properties | 21 --- .../service/src/main/resources/log4j.properties | 33 ---- .../mysql/service/src/main/resources/log4j2.xml | 35 ---- .../service/receiver/SqlReceiver.scala | 28 --- .../server/src/main/resources/log4j.properties | 33 ---- .../server/src/main/resources/log4j2.xml | 35 ---- pom.xml | 8 + .../entrance/executer/JDBCEngineExecutor.scala | 10 +- .../entrance/parser/AbstractEntranceParser.java | 5 + .../interceptor/impl/CommentInterceptor.scala | 1 + 55 files changed, 525 insertions(+), 1035 deletions(-) diff --git a/Contributing_CN.md b/Contributing_CN.md new file mode 100644 index 0000000000..f60d682355 --- /dev/null +++ b/Contributing_CN.md @@ -0,0 +1,172 @@ +# Contributing + +| **版本管理信息表** | | +| ----------- | --------------------------------- | +| 文档所属目录 | Contributing/Contributing.md | +| 现行版本 | 1.0 版，2020年3月 | +| 现行版本发布日期 | 2020 年 3 月 6 日 | +| 修订信息 | 1. 第一次发布 Linkis 项目Contributing规范。 | + +非常感谢贡献Linkis项目！在参与贡献之前，请仔细阅读以下指引。 + +## 一、贡献范畴 + +### 1.1 Bug 反馈与修复 + +我们建议无论是 Bug 反馈还是修复，都先创建一个 Issue 来仔细描述 Bug 的状况，以助于社区可以通过 Issue 记录来找到和回顾问题以及代码。Bug 反馈 Issue 通常需要包含**完整描述 Bug 的信息**以及**可复现的场景**，这样社区才能快速定位导致 Bug 的原因并修复它。包含 `#bug` 标签的打开的 Issue 都是需要被修复的。 + +### 1.2 功能交流、实现、重构 + +在交流过程中，详细描述新功能（或重构）的细节、机制和使用场景，能够促使它更好更快地被实现。**如果计划实现一个重大的功能（或重构），请务必通过 Issue 或其他方式与核心开发团队进行沟通**，这样大家能以最效率的方式来推进它。包含 `#feature` 标签的打开的 Issue 都是需要被实现的新功能，包含 `#enhancement` 标签打开的 Issue 都是需要改进重构的功能。 + +### 1.3 Issue 答疑 + +帮助回答 Issue 中的使用问题是为 Linkis 社区做贡献的一个非常有价值的方式；社区中总会有新用户不断进来，在帮助新用户的同时，也可以展现你的专业知识。 + +### 1.4 文档改进 + +Linkis 用户手册文档在 docs/ 目录下，我们使用了 [jekyll](https://jekyllrb.com/) 作为 Linkis 的文档服务，可以编辑目录里的 Markdown 文件来对文档做改进。 + +--- + +## 二、贡献流程 + +### 2.1 分支结构 + +Linkis 源码可能会产生一些临时分支，但真正有明确意义的只有以下三个分支： + +- master: 最近一次稳定 release 的源码，偶尔会多几次 hotfix 提交； +- branch-*: 最新稳定版； +- dev-*: 主要开发分支； + +请注意：大特性的dev分支，在命名时除了版本号，还会加上相应的命名说明，如：dev-0.9.4-flink，指0.9.4的flink特性开发分支。 + +### 2.2 开发指引 + +Linkis 前后端代码共用同一个代码库，但在开发上是分离的。在着手开发之前，请先将 Linkis 项目 fork 一份到自己的 Github Repositories 中， 开发时请基于自己 Github Repositories 中的 Linkis 代码库进行开发。 + +我们建议克隆 dev-* 分支来开发，这样在向 Linkis 主项目提交 PR 时合并冲突的可能性会小很多 + +```bash +git clone https://github.com/yourname/Linkis.git --branch dev-* +``` + +#### 2.2.1 后端 + +用户配置在项目根目录 /config/ 下，项目启动脚本和升级补丁脚本在项目根目录 /bin/ 下， 后端代码及核心配置在 server/ 目录下, 日志在项目根目录 /log/ 下。注意：此处所指项目根目录都指环境变量 LINKIS_HOME 所配置的目录，在使用 IDE 开发过程中也需要配置环境变量，如 Idea 关于环境变量加载的优先级：`Run/Debug Configurations` 中配置的 `Environment variables` —> IDE缓存的系统环境变量。 + +##### 2.2.1.1 目录结构 + +1. 脚本 + +``` +├── bin # 脚本目录 + ├── install.sh # 一键部署脚本 + ├── start-all.sh # 一键启动脚本 + └── stop-all.sh # 一键停止脚本 +``` + +2. 配置 + +``` +├── config # 用户配置目录 + ├── config.sh # 一键部署的配置文件 + ├── db.sh # 一键部署的数据库配置 +``` + +3. 代码目录结构 + + 详见 [Linkis代码目录结构](https://github.com/WeBankFinTech/Linkis/wiki/Linkis%E6%BA%90%E7%A0%81%E5%B1%82%E7%BA%A7%E7%BB%93%E6%9E%84%E8%AF%A6%E8%A7%A3) + +4. 日志目录 + +``` +├── logs # 日志根目录 +``` + +##### 2.2.1.2 环境变量 + + 配置系统环境变量或 IDE 环境变量 LINKIS_HOME，推荐优先使用 IDE 环境变量。 + +##### 2.2.1.3 数据库 + +1. 自行创建 Linkis 系统数据库； +2. 修改 conf/db.sh 中的数据库相应信息并执行bin/install.sh 或 直接在数据库客户端导入 db/linkis_*.sql。 + +##### 2.2.1.4 配置文件 + + 修改 每个微服务resources/ 目录下 `application.yml` 文件，配置相关属性。 + +##### 2.2.1.5 打包 + +1. 打完整 release 包需要修改根目录下 /assembly/src/main/assembly/assembly.xml 中相关版本信息，然后在根目录下执行: `mvn clean package` 即可； +2. 打 每个模块 的包可直接在 模块目录下执行 `mvn clean package`。 + +### 2.3 Pull Request 指引 + +- 如果你还不知道怎样向开源项目发起 PR，请参考[这篇说明](https://help.github.com/en/articles/about-pull-requests) +- 无论是 Bug 修复，还是新功能开发，请将 PR 提交到 dev-* 分支。 +- PR 和提交名称遵循 `<type>(<scope>): <subject>` 原则，详情可以参考阮一峰的 [Commit message 和 Change log 编写指南](http://www.ruanyifeng.com/blog/2016/01/commit_message_change_log.html) 这篇文章。 +- 如果 PR 中包含新功能，理应将文档更新包含在本次 PR 中。 +- 如果本次 PR 尚未准备好合并，请在名称头部加上 [WIP] 前缀（WIP = work-in-progress）。 +- 所有提交到 dev-* 分支的提交至少需要经过一次 Review 才可以被合并 + +### 2.4 Review 标准 + +在贡献代码之前，可以了解一下什么样的提交在 Review 中是受欢迎的。简单来说，如果一项提交能带来尽可能多增益和尽可能少的副作用或风险，那它被合并的几率就越高，Review 的速度也会越快。风险大、价值低的提交是几乎不可能被合并的，并且有可能会被拒绝连 Review 的机会都没有。 + +#### 2.4.1 增益 + +- 修复导致 Bug 的主要原因 +- 添加或修复一个大量用户亟需的功能或问题 +- 简单有效 +- 容易测试，有测试用例 +- 减少复杂度以及代码量 +- 经社区讨论过的、确定需要改进的问题 + +#### 2.4.2 副作用和风险 + +- 仅仅修复 Bug 的表面现象 +- 引入复杂度高的新功能 +- 为满足小众需求添加复杂度 +- 改动稳定的现有API或语义 +- 导致其他功能不能正常运行 +- 添加大量依赖 +- 随意改变依赖版本 +- 一次性提交大量代码或改动 + +#### 2.4.3 Reviewer 注意事项 + +- 请使用建设性语气撰写评论 +- 如果需要提交者进行修改，请明确说明完成此次 Pull Request 所需要修改的所有内容 +- 如果某次 PR 在合并后发现带来了新问题，Reviewer 需要向 PR 作者联系并沟通解决问题；如果无法联系到 PR 作者，Reviewer 需要将此次 PR 进行还原 + +--- + +## 三、贡献进阶 + +### 3.1 关于 Committers（Collaborators） + +#### 3.1.1 如何成为 Committer + +如果你对 Linkis 提过颇具价值的 PR 并且被合并，或是连续贡献超过半年，且至少主导过一次版本的发布，你可以通过官方微信群找到Linkis项目的一个 PMC ，如果他愿意提名你为 committer，并愿意为你陈述你的贡献给所有 PMC和Committer，那么接下来会发起一次投票；PMC和其他 Committers 将会一起投票决定是否允许你的加入，如果得到足够票数，你将成为 Linkis 项目的 Committer。 + +#### 3.1.2 Committer 的权利 + +- 可以加入官方开发者微信群，参与讨论和制定Linkis开发计划 +- 可以对 Issue 进行管理，包括关闭、添加标签 +- 可以创建和管理项目分支，master、dev-* 分支除外 +- 可以对提交到 dev-* 分支的 PR 进行 Review +- 可以申请成为 Committee 成员 + +### 3.2 关于 Committee + +#### 3.2.1 如何成为 Committee 成员 + +如果你是 Linkis 项目的 Committer，并且你贡献的所有内容得到了其他 Committee 成员的认可，你可以申请成为 Linkis Committee 成员，其他 Committee 成员将会一起投票决定是否允许你的加入，如果全票通过，你将成为 Linkis Committee 成员。 + +#### 3.2.2 Committee 成员的权利 + +- 可以合并其他 Committers 和贡献者提交到 dev-** 分支的 PR +- 可以参与决定Linkis项目的roadmap和发展方向 +- 可以参与新版本发布 diff --git a/bml/bmlserver/conf/log4j2.xml b/bml/bmlserver/conf/log4j2.xml index 53593a323e..3da04d9e2f 100644 --- a/bml/bmlserver/conf/log4j2.xml +++ b/bml/bmlserver/conf/log4j2.xml @@ -5,8 +5,8 @@ <ThresholdFilter level="trace" onMatch="ACCEPT" onMismatch="DENY"/> <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/> </Console> - <RollingFile name="RollingFile" fileName="/appcom/logs/linkis/BmlServer/BmlServer.log" - filePattern="/appcom/logs/linkis/BmlServer/$${date:yyyy-MM}/BmlServer-log-%d{yyyy-MM-dd}-%i.log.gz"> + <RollingFile name="RollingFile" fileName="logs/BmlServer.log" + filePattern="logs/%d{yyyy-MM}/BmlServer-log-%d{yyyy-MM-dd}-%i.log.gz"> <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/> <SizeBasedTriggeringPolicy size="100MB"/> <DefaultRolloverStrategy max="20"/> diff --git a/datasource/datasourcemanager/server/bin/start-dsm.sh b/datasource/datasourcemanager/server/bin/start-dsm.sh new file mode 100644 index 0000000000..80cc775a4a --- /dev/null +++ b/datasource/datasourcemanager/server/bin/start-dsm.sh @@ -0,0 +1,38 @@ +#!/bin/bash + +cd `dirname $0` +cd .. +HOME=`pwd` + +export SERVER_PID=$HOME/bin/linkis.pid +export SERVER_LOG_PATH=$HOME/logs +export SERVER_CLASS=com.webank.wedatasphere.linkis.DataWorkCloudApplication + +if test -z "$SERVER_HEAP_SIZE" +then + export SERVER_HEAP_SIZE="512M" +fi + +if test -z "$SERVER_JAVA_OPTS" +then + export SERVER_JAVA_OPTS=" -Xmx$SERVER_HEAP_SIZE -XX:+UseG1GC -Xloggc:$HOME/logs/linkis-gc.log" +fi + +if [[ -f "${SERVER_PID}" ]]; then + pid=$(cat ${SERVER_PID}) + if kill -0 ${pid} >/dev/null 2>&1; then + echo "Server is already running." + exit 1 + fi +fi + +nohup java $SERVER_JAVA_OPTS -cp ../module/lib/*:$HOME/conf:$HOME/lib/* $SERVER_CLASS 2>&1 > $SERVER_LOG_PATH/linkis.out & +pid=$! +if [[ -z "${pid}" ]]; then + echo "server $SERVER_NAME start failed!" + exit 1 +else + echo "server $SERVER_NAME start succeeded!" + echo $pid > $SERVER_PID + sleep 1 +fi \ No newline at end of file diff --git a/datasource/datasourcemanager/server/bin/stop-dsm.sh b/datasource/datasourcemanager/server/bin/stop-dsm.sh new file mode 100644 index 0000000000..f032887111 --- /dev/null +++ b/datasource/datasourcemanager/server/bin/stop-dsm.sh @@ -0,0 +1,47 @@ +#!/bin/bash + +cd `dirname $0` +cd .. +HOME=`pwd` + +export SERVER_PID=$HOME/bin/linkis.pid + +function wait_for_server_to_die() { + local pid + local count + pid=$1 + timeout=$2 + count=0 + timeoutTime=$(date "+%s") + let "timeoutTime+=$timeout" + currentTime=$(date "+%s") + forceKill=1 + + while [[ $currentTime -lt $timeoutTime ]]; do + $(kill ${pid} > /dev/null 2> /dev/null) + if kill -0 ${pid} > /dev/null 2>&1; then + sleep 3 + else + forceKill=0 + break + fi + currentTime=$(date "+%s") + done + + if [[ forceKill -ne 0 ]]; then + $(kill -9 ${pid} > /dev/null 2> /dev/null) + fi +} + +if [[ ! -f "${SERVER_PID}" ]]; then + echo "server $SERVER_NAME is not running" +else + pid=$(cat ${SERVER_PID}) + if [[ -z "${pid}" ]]; then + echo "server $SERVER_NAME is not running" + else + wait_for_server_to_die $pid 40 + $(rm -f ${SERVER_PID}) + echo "server $SERVER_NAME is stopped." + fi +fi \ No newline at end of file diff --git a/datasourcemanager/server/src/main/resources/application.yml b/datasource/datasourcemanager/server/conf/application.yml similarity index 100% rename from datasourcemanager/server/src/main/resources/application.yml rename to datasource/datasourcemanager/server/conf/application.yml diff --git a/datasourcemanager/server/src/main/resources/linkis.properties b/datasource/datasourcemanager/server/conf/linkis.properties similarity index 100% rename from datasourcemanager/server/src/main/resources/linkis.properties rename to datasource/datasourcemanager/server/conf/linkis.properties diff --git a/datasource/metadatamanager/elasticsearch/src/main/resources/log4j.properties b/datasource/datasourcemanager/server/conf/log4j.properties similarity index 100% rename from datasource/metadatamanager/elasticsearch/src/main/resources/log4j.properties rename to datasource/datasourcemanager/server/conf/log4j.properties diff --git a/datasource/metadatamanager/elasticsearch/src/main/resources/log4j2.xml b/datasource/datasourcemanager/server/conf/log4j2.xml similarity index 100% rename from datasource/metadatamanager/elasticsearch/src/main/resources/log4j2.xml rename to datasource/datasourcemanager/server/conf/log4j2.xml diff --git a/datasourcemanager/server/pom.xml b/datasource/datasourcemanager/server/pom.xml similarity index 77% rename from datasourcemanager/server/pom.xml rename to datasource/datasourcemanager/server/pom.xml index a20439d415..d656deef24 100644 --- a/datasourcemanager/server/pom.xml +++ b/datasource/datasourcemanager/server/pom.xml @@ -114,17 +114,52 @@ <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-assembly-plugin</artifactId> + <version>2.3</version> + <inherited>false</inherited> + <executions> + <execution> + <id>make-assembly</id> + <phase>package</phase> + <goals> + <goal>single</goal> + </goals> + <configuration> + <descriptors> + <descriptor>src/main/assembly/distribution.xml</descriptor> + </descriptors> + </configuration> + </execution> + </executions> + <configuration> + <skipAssembly>false</skipAssembly> + <finalName>linkis-dsm-server</finalName> + <appendAssemblyId>false</appendAssemblyId> + <attach>false</attach> + <descriptors> + <descriptor>src/main/assembly/distribution.xml</descriptor> + </descriptors> + </configuration> + </plugin> </plugins> <resources> - <resource> - <directory>src/main/resources</directory> - </resource> <resource> <directory>src/main/java</directory> <includes> <include>**/*.xml</include> </includes> </resource> + <resource> + <directory>src/main/resources</directory> + <excludes> + <exclude>**/*.properties</exclude> + <exclude>**/application.yml</exclude> + <exclude>**/bootstrap.yml</exclude> + <exclude>**/log4j2.xml</exclude> + </excludes> + </resource> </resources> <finalName>${project.artifactId}-${project.version}</finalName> </build> diff --git a/datasource/datasourcemanager/server/src/main/assembly/distribution.xml b/datasource/datasourcemanager/server/src/main/assembly/distribution.xml new file mode 100644 index 0000000000..4ba803e71d --- /dev/null +++ b/datasource/datasourcemanager/server/src/main/assembly/distribution.xml @@ -0,0 +1,206 @@ +<!-- + ~ Copyright 2019 WeBank + ~ + ~ Licensed under the Apache License, Version 2.0 (the "License"); + ~ you may not use this file except in compliance with the License. + ~ You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + --> + +<assembly + xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/2.3"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/2.3 http://maven.apache.org/xsd/assembly-1.1.2.xsd";> + <id>linkis-dsm-server</id> + <formats> + <format>zip</format> + </formats> + <includeBaseDirectory>true</includeBaseDirectory> + <baseDirectory>linkis-dsm-server</baseDirectory> + + <dependencySets> + <dependencySet> + <!-- Enable access to all projects in the current multimodule build! <useAllReactorProjects>true</useAllReactorProjects> --> + <!-- Now, select which projects to include in this module-set. --> + <outputDirectory>lib</outputDirectory> + <useProjectArtifact>true</useProjectArtifact> + <useTransitiveDependencies>true</useTransitiveDependencies> + <unpack>false</unpack> + <useStrictFiltering>true</useStrictFiltering> + <useTransitiveFiltering>true</useTransitiveFiltering> + + <!-- <excludes> + <exclude>antlr:antlr:jar</exclude> + <exclude>aopalliance:aopalliance:jar</exclude> + <exclude>com.fasterxml.jackson.core:jackson-annotations:jar</exclude> + <exclude>com.fasterxml.jackson.core:jackson-core:jar</exclude> + <exclude>com.fasterxml.jackson.core:jackson-databind:jar</exclude> + <exclude>com.fasterxml.jackson.module:jackson-module-paranamer:jar</exclude> + <exclude>com.fasterxml.jackson.module:jackson-module-scala_2.11:jar</exclude> + <exclude>com.google.code.gson:gson:jar</exclude> + <exclude>com.google.guava:guava:jar</exclude> + <exclude>com.google.protobuf:protobuf-java:jar</exclude> + <exclude>com.netflix.archaius:archaius-core:jar</exclude> + <exclude>com.netflix.hystrix:hystrix-core:jar</exclude> + <exclude>com.netflix.netflix-commons:netflix-commons-util:jar</exclude> + <exclude>com.netflix.netflix-commons:netflix-statistics:jar</exclude> + <exclude>com.netflix.ribbon:ribbon:jar</exclude> + <exclude>com.netflix.ribbon:ribbon-core:jar</exclude> + <exclude>com.netflix.ribbon:ribbon-httpclient:jar</exclude> + <exclude>com.netflix.ribbon:ribbon-loadbalancer:jar</exclude> + <exclude>com.netflix.ribbon:ribbon-transport:jar</exclude> + <exclude>com.netflix.servo:servo-core:jar</exclude> + <exclude>com.sun.jersey.contribs:jersey-apache-client4:jar</exclude> + <exclude>com.sun.jersey:jersey-client:jar</exclude> + <exclude>com.sun.jersey:jersey-core:jar</exclude> + <exclude>com.sun.jersey:jersey-server:jar</exclude> + <exclude>com.sun.xml.bind:jaxb-impl:jar</exclude> + <exclude>com.webank.wedatasphere.linkis:linkis-common:jar</exclude> + <exclude>commons-beanutils:commons-beanutils:jar</exclude> + <exclude>commons-beanutils:commons-beanutils-core:jar</exclude> + <exclude>commons-cli:commons-cli:jar</exclude> + <exclude>commons-collections:commons-collections:jar</exclude> + <exclude>commons-configuration:commons-configuration:jar</exclude> + <exclude>commons-daemon:commons-daemon:jar</exclude> + <exclude>commons-dbcp:commons-dbcp:jar</exclude> + <exclude>commons-digester:commons-digester:jar</exclude> + <exclude>commons-io:commons-io:jar</exclude> + <exclude>commons-lang:commons-lang:jar</exclude> + <exclude>commons-net:commons-net:jar</exclude> + <exclude>commons-pool:commons-pool:jar</exclude> + <exclude>io.netty:netty:jar</exclude> + <exclude>io.netty:netty-all:jar</exclude> + <exclude>io.netty:netty-buffer:jar</exclude> + <exclude>io.netty:netty-codec:jar</exclude> + <exclude>io.netty:netty-codec-http:jar</exclude> + <exclude>io.netty:netty-common:jar</exclude> + <exclude>io.netty:netty-handler:jar</exclude> + <exclude>io.netty:netty-transport:jar</exclude> + <exclude>io.netty:netty-transport-native-epoll:jar</exclude> + <exclude>io.reactivex:rxjava:jar</exclude> + <exclude>io.reactivex:rxnetty:jar</exclude> + <exclude>io.reactivex:rxnetty-contexts:jar</exclude> + <exclude>io.reactivex:rxnetty-servo:jar</exclude> + <exclude>javax.activation:activation:jar</exclude> + <exclude>javax.annotation:javax.annotation-api:jar</exclude> + <exclude>javax.inject:javax.inject:jar</exclude> + <exclude>javax.servlet:javax.servlet-api:jar</exclude> + <exclude>javax.servlet.jsp:jsp-api:jar</exclude> + <exclude>javax.xml.bind:jaxb-api:jar</exclude> + <exclude>javax.xml.stream:stax-api:jar</exclude> + <exclude>net.databinder.dispatch:dispatch-core_2.11:jar</exclude> + <exclude>net.databinder.dispatch:dispatch-json4s-jackson_2.11:jar</exclude> + <exclude>org.antlr:antlr-runtime:jar</exclude> + <exclude>org.antlr:stringtemplate:jar</exclude> + <exclude>org.apache.commons:commons-compress:jar</exclude> + <exclude>org.apache.commons:commons-math3:jar</exclude> + <exclude>org.apache.curator:curator-framework:jar</exclude> + <exclude>org.apache.curator:curator-recipes:jar</exclude> + <exclude>org.apache.directory.api:api-asn1-api:jar</exclude> + <exclude>org.apache.directory.api:api-util:jar</exclude> + <exclude>org.apache.directory.server:apacheds-i18n:jar</exclude> + <exclude>org.apache.directory.server:apacheds-kerberos-codec:jar</exclude> + <exclude>org.apache.hadoop:hadoop-annotations:jar</exclude> + <exclude>org.apache.hadoop:hadoop-auth:jar</exclude> + <exclude>org.apache.hadoop:hadoop-common:jar</exclude> + <exclude>org.apache.hadoop:hadoop-hdfs:jar</exclude> + <exclude>org.apache.htrace:htrace-core:jar</exclude> + <exclude>org.apache.logging.log4j:log4j-api:jar</exclude> + <exclude>org.apache.zookeeper:zookeeper:jar</exclude> + <exclude>org.aspectj:aspectjweaver:jar</exclude> + <exclude>org.bouncycastle:bcpkix-jdk15on:jar</exclude> + <exclude>org.bouncycastle:bcprov-jdk15on:jar</exclude> + <exclude>org.codehaus.jettison:jettison:jar</exclude> + <exclude>org.eclipse.jetty:jetty-continuation:jar</exclude> + <exclude>org.eclipse.jetty:jetty-http:jar</exclude> + <exclude>org.eclipse.jetty:jetty-io:jar</exclude> + <exclude>org.eclipse.jetty:jetty-jndi:jar</exclude> + <exclude>org.eclipse.jetty:jetty-plus:jar</exclude> + <exclude>org.eclipse.jetty:jetty-security:jar</exclude> + <exclude>org.eclipse.jetty:jetty-server:jar</exclude> + <exclude>org.eclipse.jetty:jetty-servlet:jar</exclude> + <exclude>org.eclipse.jetty:jetty-util:jar</exclude> + <exclude>org.eclipse.jetty:jetty-webapp:jar</exclude> + <exclude>org.eclipse.jetty:jetty-xml:jar</exclude> + <exclude>org.fusesource.leveldbjni:leveldbjni-all:jar</exclude> + <exclude>org.hdrhistogram:HdrHistogram:jar</exclude> + <exclude>org.json4s:json4s-ast_2.11:jar</exclude> + <exclude>org.json4s:json4s-core_2.11:jar</exclude> + <exclude>org.json4s:json4s-jackson_2.11:jar</exclude> + <exclude>org.jsoup:jsoup:jar</exclude> + <exclude>org.mortbay.jetty:jetty:jar</exclude> + <exclude>org.mortbay.jetty:jetty-util:jar</exclude> + <exclude>org.scala-lang.modules:scala-parser-combinators_2.11:jar</exclude> + <exclude>org.scala-lang.modules:scala-xml_2.11:jar</exclude> + <exclude>org.scala-lang:scala-compiler:jar</exclude> + <exclude>org.scala-lang:scala-library:jar</exclude> + <exclude>org.scala-lang:scala-reflect:jar</exclude> + <exclude>org.scala-lang:scalap:jar</exclude> + <exclude>org.slf4j:jul-to-slf4j:jar</exclude> + <exclude>org.slf4j:slf4j-api:jar</exclude> + <exclude>org.springframework.boot:spring-boot:jar</exclude> + <exclude>org.springframework.boot:spring-boot-autoconfigure:jar</exclude> + <exclude>org.springframework.boot:spring-boot-starter:jar</exclude> + <exclude>org.springframework.boot:spring-boot-starter-aop:jar</exclude> + <exclude>org.springframework.cloud:spring-cloud-commons:jar</exclude> + <exclude>org.springframework.cloud:spring-cloud-context:jar</exclude> + <exclude>org.springframework.cloud:spring-cloud-netflix-archaius:jar</exclude> + <exclude>org.springframework.cloud:spring-cloud-netflix-ribbon:jar</exclude> + <exclude>org.springframework.cloud:spring-cloud-starter:jar</exclude> + <exclude>org.springframework.cloud:spring-cloud-starter-netflix-archaius:jar</exclude> + <exclude>org.springframework.cloud:spring-cloud-starter-netflix-ribbon:jar</exclude> + <exclude>org.springframework.security:spring-security-crypto:jar</exclude> + <exclude>org.springframework.security:spring-security-rsa:jar</exclude> + <exclude>org.springframework:spring-aop:jar</exclude> + <exclude>org.springframework:spring-beans:jar</exclude> + <exclude>org.springframework:spring-context:jar</exclude> + <exclude>org.springframework:spring-core:jar</exclude> + <exclude>org.springframework:spring-expression:jar</exclude> + <!–<exclude>org.springframework:spring-jcl:jar</exclude>–> + <exclude>org.springframework:spring-web:jar</exclude> + <exclude>org.tukaani:xz:jar</exclude> + <exclude>org.yaml:snakeyaml:jar</exclude> + <exclude>xerces:xercesImpl:jar</exclude> + <exclude>xml-apis:xml-apis:jar</exclude> + <exclude>xmlenc:xmlenc:jar</exclude> + </excludes>--> + </dependencySet> + </dependencySets> + + <fileSets> + <fileSet> + <directory>${basedir}/conf</directory> + <includes> + <include>*</include> + </includes> + <fileMode>0777</fileMode> + <outputDirectory>conf</outputDirectory> + <lineEnding>unix</lineEnding> + </fileSet> + <fileSet> + <directory>${basedir}/bin</directory> + <includes> + <include>*</include> + </includes> + <fileMode>0777</fileMode> + <outputDirectory>bin</outputDirectory> + <lineEnding>unix</lineEnding> + </fileSet> + <fileSet> + <directory>.</directory> + <excludes> + <exclude>*/**</exclude> + </excludes> + <outputDirectory>logs</outputDirectory> + </fileSet> + </fileSets> + +</assembly> + diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/CustomMultiPartFormDataTransformer.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/CustomMultiPartFormDataTransformer.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/CustomMultiPartFormDataTransformer.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/CustomMultiPartFormDataTransformer.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormDataTransformerFactory.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormDataTransformerFactory.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormDataTransformerFactory.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormDataTransformerFactory.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormStreamContent.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormStreamContent.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormStreamContent.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/FormStreamContent.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/MultiPartFormDataTransformer.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/MultiPartFormDataTransformer.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/MultiPartFormDataTransformer.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/formdata/MultiPartFormDataTransformer.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/BmlAppService.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/BmlAppService.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/BmlAppService.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/BmlAppService.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceInfoService.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceInfoService.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceInfoService.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceInfoService.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceOpService.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceOpService.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceOpService.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceOpService.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceRelateService.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceRelateService.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceRelateService.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/DataSourceRelateService.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/MetadataOperateService.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/MetadataOperateService.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/MetadataOperateService.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/MetadataOperateService.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/BmlAppServiceImpl.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/BmlAppServiceImpl.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/BmlAppServiceImpl.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/BmlAppServiceImpl.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceInfoServiceImpl.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceRelateServiceImpl.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceRelateServiceImpl.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceRelateServiceImpl.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/DataSourceRelateServiceImpl.java diff --git a/datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/MetadataOperateServiceImpl.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/MetadataOperateServiceImpl.java similarity index 100% rename from datasource/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/MetadataOperateServiceImpl.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/service/impl/MetadataOperateServiceImpl.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/DataSourceParameterValidator.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/DataSourceParameterValidator.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/DataSourceParameterValidator.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/DataSourceParameterValidator.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateException.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateException.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateException.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateException.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateStrategy.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateStrategy.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateStrategy.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidateStrategy.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidator.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidator.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidator.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/ParameterValidator.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/RegExpParameterValidateStrategy.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/RegExpParameterValidateStrategy.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/RegExpParameterValidateStrategy.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/RegExpParameterValidateStrategy.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/TypeParameterValidateStrategy.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/TypeParameterValidateStrategy.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/TypeParameterValidateStrategy.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/validate/strategy/TypeParameterValidateStrategy.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceEnvVo.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceEnvVo.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceEnvVo.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceEnvVo.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceVo.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceVo.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceVo.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/DataSourceVo.java diff --git a/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/PageViewVo.java b/datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/PageViewVo.java similarity index 100% rename from datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/PageViewVo.java rename to datasource/datasourcemanager/server/src/main/java/com/webank/wedatasphere/linkis/datasourcemanager/core/vo/PageViewVo.java diff --git a/datasourcemanager/server/src/main/scala/com/webank/wedatasphere/linkis/datasourcemanager/core/receivers/DsmReceiver.scala b/datasource/datasourcemanager/server/src/main/scala/com/webank/wedatasphere/linkis/datasourcemanager/core/receivers/DsmReceiver.scala similarity index 100% rename from datasourcemanager/server/src/main/scala/com/webank/wedatasphere/linkis/datasourcemanager/core/receivers/DsmReceiver.scala rename to datasource/datasourcemanager/server/src/main/scala/com/webank/wedatasphere/linkis/datasourcemanager/core/receivers/DsmReceiver.scala diff --git a/datasource/metadatamanager/elasticsearch/pom.xml b/datasource/metadatamanager/elasticsearch/pom.xml deleted file mode 100644 index 21d5c0759c..0000000000 --- a/datasource/metadatamanager/elasticsearch/pom.xml +++ /dev/null @@ -1,78 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> - -<!-- - ~ Copyright 2019 WeBank - ~ Licensed under the Apache License, Version 2.0 (the "License"); - ~ you may not use this file except in compliance with the License. - ~ You may obtain a copy of the License at - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> - -<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> - <parent> - <artifactId>linkis</artifactId> - <groupId>com.webank.wedatasphere.linkis</groupId> - <version>0.9.4</version> - </parent> - <modelVersion>4.0.0</modelVersion> - - <artifactId>linkis-metadatamanager-service-es</artifactId> - <properties> - <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> - <es.version>6.7.1</es.version> - </properties> - - <dependencies> - <dependency> - <groupId>com.webank.wedatasphere.linkis</groupId> - <artifactId>linkis-metadatamanager-common</artifactId> - <version>${linkis.version}</version> - </dependency> - <dependency> - <groupId>com.webank.wedatasphere.linkis</groupId> - <artifactId>linkis-module</artifactId> - <version>${linkis.version}</version> - <exclusions> - <exclusion> - <artifactId>asm</artifactId> - <groupId>org.ow2.asm</groupId> - </exclusion> - </exclusions> - </dependency> - <dependency> - <groupId>org.elasticsearch.client</groupId> - <artifactId>elasticsearch-rest-client</artifactId> - <version>${es.version}</version> - </dependency> - </dependencies> - - <build> - <plugins> - <plugin> - <groupId>org.apache.maven.plugins</groupId> - <artifactId>maven-deploy-plugin</artifactId> - </plugin> - - <plugin> - <groupId>net.alchim31.maven</groupId> - <artifactId>scala-maven-plugin</artifactId> - </plugin> - <plugin> - <groupId>org.apache.maven.plugins</groupId> - <artifactId>maven-jar-plugin</artifactId> - </plugin> - </plugins> - <resources> - <resource> - <directory>src/main/resources</directory> - </resource> - </resources> - <finalName>${project.artifactId}-${project.version}</finalName> - </build> -</project> diff --git a/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticConnection.java b/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticConnection.java deleted file mode 100644 index b485c69a3a..0000000000 --- a/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticConnection.java +++ /dev/null @@ -1,140 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service; - -import com.webank.wedatasphere.linkis.metadatamanager.common.Json; -import org.apache.commons.lang.StringUtils; -import org.apache.http.HttpHost; -import org.apache.http.auth.AuthScope; -import org.apache.http.auth.UsernamePasswordCredentials; -import org.apache.http.client.CredentialsProvider; -import org.apache.http.impl.client.BasicCredentialsProvider; -import org.apache.http.impl.nio.reactor.IOReactorConfig; -import org.elasticsearch.client.Request; -import org.elasticsearch.client.Response; -import org.elasticsearch.client.RestClient; -import org.elasticsearch.client.RestClientBuilder; - -import java.io.Closeable; -import java.io.IOException; -import java.util.ArrayList; -import java.util.List; -import java.util.Map; -import java.util.Set; - -/** - * @author davidhua - * 2020/02/14 - */ -public class ElasticConnection implements Closeable { - - public static final String DEFAULT_TYPE_NAME = "type"; - - private static final String DEFAULT_MAPPING_NAME = "mappings"; - private static final String DEFAULT_INDEX_NAME = "index"; - private static final String FIELD_PROPS = "properties"; - - private RestClient restClient; - - public ElasticConnection(String[] endPoints, String username, String password) throws IOException { - HttpHost[] httpHosts = new HttpHost[endPoints.length]; - for(int i = 0; i < endPoints.length; i++){ - httpHosts[i] = HttpHost.create(endPoints[i]); - } - RestClientBuilder restClientBuilder = RestClient.builder(httpHosts); - CredentialsProvider credentialsProvider = null; - if(StringUtils.isNotBlank(username) && StringUtils.isNotBlank(password)){ - credentialsProvider = new BasicCredentialsProvider(); - credentialsProvider.setCredentials(AuthScope.ANY, - new UsernamePasswordCredentials(username, password)); - } - //set only one thread - CredentialsProvider finalCredentialsProvider = credentialsProvider; - restClientBuilder.setHttpClientConfigCallback(httpClientBuilder -> { - if(null != finalCredentialsProvider){ - httpClientBuilder - .setDefaultCredentialsProvider(finalCredentialsProvider); - } - return httpClientBuilder.setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(1).build()); - }); - this.restClient = restClientBuilder.build(); - //Try to test connection - ping(); - } - - public List<String> getAllIndices() throws IOException { - List<String> indices = new ArrayList<>(); - Request request = new Request("GET", "_cat/indices"); - request.addParameter("format", "JSON"); - Response response = restClient.performRequest(request); - List<Map<String, Object>> list = Json.fromJson(response.getEntity().getContent(), Map.class); - list.forEach( v ->{ - String index = String.valueOf(v.getOrDefault(DEFAULT_INDEX_NAME, "")); - if(StringUtils.isNotBlank(index) && !index.startsWith(".")){ - indices.add(index); - } - }); - return indices; - } - - public List<String> getTypes(String index) throws IOException{ - List<String> types = new ArrayList<>(); - Request request = new Request("GET", index +"/_mappings"); - Response response = restClient.performRequest(request); - Map<String, Map<String, Object>> result = - Json.fromJson(response.getEntity().getContent(), Map.class); - Map<String, Object> indexMap = result.get(index); - Object props = indexMap.get(DEFAULT_MAPPING_NAME); - if(props instanceof Map){ - Set keySet = ((Map)props).keySet(); - for(Object v : keySet){ - types.add(String.valueOf(v)); - } - } - return types; - } - - public Map<Object, Object> getProps(String index, String type) throws IOException{ - Request request = new Request("GET", index + "/_mappings/" + type); - Response response = restClient.performRequest(request); - Map<String, Map<String, Object>> result = - Json.fromJson(response.getEntity().getContent(), Map.class); - Map mappings = (Map)result.get(index).get("mappings"); - Map propsMap = mappings; - if(mappings.containsKey(type)){ - Object typeMap = mappings.get(type); - if(typeMap instanceof Map){ - propsMap = (Map)typeMap; - } - } - Object props = propsMap.get(FIELD_PROPS); - if(props instanceof Map){ - return (Map)props; - } - return null; - } - public void ping() throws IOException{ - Response response = restClient.performRequest(new Request("GET", "/")); - int code = response.getStatusLine().getStatusCode(); - int successCode = 200; - if(code != successCode){ - throw new RuntimeException("Ping to ElasticSearch ERROR, response code: " + code); - } - } - - @Override - public void close() throws IOException { - this.restClient.close(); - } -} diff --git a/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticMetaService.java b/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticMetaService.java deleted file mode 100644 index 8778ba42eb..0000000000 --- a/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticMetaService.java +++ /dev/null @@ -1,87 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service; - -import com.webank.wedatasphere.linkis.metadatamanager.common.Json; -import com.webank.wedatasphere.linkis.metadatamanager.common.domain.MetaColumnInfo; -import com.webank.wedatasphere.linkis.metadatamanager.common.service.AbstractMetaService; -import com.webank.wedatasphere.linkis.metadatamanager.common.service.MetadataConnection; -import org.springframework.stereotype.Component; - -import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; - -/** - * @author davidhua - * 2020/02/14 - */ -@Component -public class ElasticMetaService extends AbstractMetaService<ElasticConnection> { - @Override - public MetadataConnection<ElasticConnection> getConnection(String operator, Map<String, Object> params) - throws Exception{ - String[] endPoints = new String[]{}; - Object urls = params.get(ElasticParamsMapper.PARAM_ES_URLS.getValue()); - if(!(urls instanceof List)){ - List<String> urlList = Json.fromJson(String.valueOf(urls), List.class, String.class); - assert urlList != null; - endPoints = urlList.toArray(endPoints); - }else{ - endPoints = ((List<String>)urls).toArray(endPoints); - } - ElasticConnection conn = new ElasticConnection(endPoints, String.valueOf(params.getOrDefault(ElasticParamsMapper.PARAM_ES_USERNAME.getValue(), "")), - String.valueOf(params.getOrDefault(ElasticParamsMapper.PARAM_ES_PASSWORD.getValue(), ""))); - return new MetadataConnection<>(conn, false); - } - - @Override - public List<String> queryDatabases(ElasticConnection connection) { - //Get indices - try{ - return connection.getAllIndices(); - }catch (Exception e){ - throw new RuntimeException("Fail to get ElasticSearch indices(获取索引列表失败)", e); - } - } - - @Override - public List<String> queryTables(ElasticConnection connection, String database) { - //Get types - try{ - return connection.getTypes(database); - }catch (Exception e){ - throw new RuntimeException("Fail to get ElasticSearch types(获取索引类型失败)", e); - } - } - - @Override - public List<MetaColumnInfo> queryColumns(ElasticConnection connection, String database, String table) { - try { - Map<Object, Object> props = connection.getProps(database, table); - return props.entrySet().stream().map(entry -> { - MetaColumnInfo info = new MetaColumnInfo(); - info.setName(String.valueOf(entry.getKey())); - Object value = entry.getValue(); - if(value instanceof Map){ - info.setType(String.valueOf(((Map)value) - .getOrDefault(ElasticConnection.DEFAULT_TYPE_NAME, ""))); - } - return info; - }).collect(Collectors.toList()); - } catch (Exception e) { - throw new RuntimeException("Fail to get ElasticSearch columns(获取索引字段失败)", e); - } - } -} diff --git a/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticParamsMapper.java b/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticParamsMapper.java deleted file mode 100644 index 3af7e8ad86..0000000000 --- a/datasource/metadatamanager/elasticsearch/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/ElasticParamsMapper.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service; - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; - -/** - * Configuration - * @author davidhua - * 2020/02/14 - */ -public class ElasticParamsMapper { - public static final CommonVars<String> PARAM_ES_URLS = - CommonVars.apply("wds.linkis.server.mdm.service.es.urls", "elasticUrls"); - - public static final CommonVars<String> PARAM_ES_USERNAME = - CommonVars.apply("wds.linkis.server.mdm.service.es.username", "username"); - - public static final CommonVars<String> PARAM_ES_PASSWORD = - CommonVars.apply("wds.linkis.server.mdm.service.es.password", "password"); -} diff --git a/datasource/metadatamanager/elasticsearch/src/main/resources/application.yml b/datasource/metadatamanager/elasticsearch/src/main/resources/application.yml deleted file mode 100644 index 8b3f9ce44a..0000000000 --- a/datasource/metadatamanager/elasticsearch/src/main/resources/application.yml +++ /dev/null @@ -1,23 +0,0 @@ -server: - port: 8295 -spring: - application: - name: mdm-service-elasticsearch - -eureka: - client: - serviceUrl: - defaultZone: http://${ip}:${port}/eureka/ - instance: - metadata-map: - test: wedatasphere - -management: - endpoints: - web: - exposure: - include: refresh,info -logging: - config: classpath:log4j2.xml - - diff --git a/datasource/metadatamanager/elasticsearch/src/main/resources/linkis.properties b/datasource/metadatamanager/elasticsearch/src/main/resources/linkis.properties deleted file mode 100644 index f83c10d281..0000000000 --- a/datasource/metadatamanager/elasticsearch/src/main/resources/linkis.properties +++ /dev/null @@ -1,21 +0,0 @@ -# -# Copyright 2019 WeBank -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -wds.linkis.server.mybatis.mapperLocations= -wds.linkis.server.mybatis.typeAliasesPackage= -wds.linkis.server.mybatis.BasePackage= -wds.linkis.server.restful.scan.packages= - -#sit -wds.linkis.server.version=v1 - diff --git a/datasource/metadatamanager/elasticsearch/src/main/scala/com/webank/wedatasphere/linkis/metadatamanager/service/receiver/ElasticReceiver.scala b/datasource/metadatamanager/elasticsearch/src/main/scala/com/webank/wedatasphere/linkis/metadatamanager/service/receiver/ElasticReceiver.scala deleted file mode 100644 index dd3049f1de..0000000000 --- a/datasource/metadatamanager/elasticsearch/src/main/scala/com/webank/wedatasphere/linkis/metadatamanager/service/receiver/ElasticReceiver.scala +++ /dev/null @@ -1,29 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service.receiver - -import com.webank.wedatasphere.linkis.DataWorkCloudApplication -import com.webank.wedatasphere.linkis.metadatamanager.common.service.MetadataService -import com.webank.wedatasphere.linkis.metadatamanager.common.receiver.BaseMetaReceiver -import com.webank.wedatasphere.linkis.metadatamanager.common.service.MetadataService -import javax.annotation.PostConstruct -import org.springframework.stereotype.Component - -@Component -class ElasticReceiver extends BaseMetaReceiver{ - @PostConstruct - def init(): Unit = { - metadataService = DataWorkCloudApplication.getApplicationContext.getBean(classOf[MetadataService]) - } -} diff --git a/datasource/metadatamanager/mysql/service/pom.xml b/datasource/metadatamanager/mysql/service/pom.xml deleted file mode 100644 index 65acc14528..0000000000 --- a/datasource/metadatamanager/mysql/service/pom.xml +++ /dev/null @@ -1,83 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> - -<!-- - ~ Copyright 2019 WeBank - ~ Licensed under the Apache License, Version 2.0 (the "License"); - ~ you may not use this file except in compliance with the License. - ~ You may obtain a copy of the License at - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> - -<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> - <parent> - <artifactId>linkis</artifactId> - <groupId>com.webank.wedatasphere.linkis</groupId> - <version>0.9.4</version> - </parent> - <modelVersion>4.0.0</modelVersion> - - <artifactId>linkis-metadatamanager-service-mysql</artifactId> - - <properties> - <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> - <mysql.version>5.1.34</mysql.version> - </properties> - - <dependencies> - <dependency> - <groupId>com.webank.wedatasphere.linkis</groupId> - <artifactId>linkis-metadatamanager-common</artifactId> - <version>${linkis.version}</version> - </dependency> - <dependency> - <groupId>com.webank.wedatasphere.linkis</groupId> - <artifactId>linkis-module</artifactId> - <version>${linkis.version}</version> - <exclusions> - <exclusion> - <artifactId>asm</artifactId> - <groupId>org.ow2.asm</groupId> - </exclusion> - <exclusion> - <groupId>mysql</groupId> - <artifactId>mysql-connector-java</artifactId> - </exclusion> - </exclusions> - </dependency> - <dependency> - <groupId>mysql</groupId> - <artifactId>mysql-connector-java</artifactId> - <version>${mysql.version}</version> - </dependency> - </dependencies> - - <build> - <plugins> - <plugin> - <groupId>org.apache.maven.plugins</groupId> - <artifactId>maven-deploy-plugin</artifactId> - </plugin> - - <plugin> - <groupId>net.alchim31.maven</groupId> - <artifactId>scala-maven-plugin</artifactId> - </plugin> - <plugin> - <groupId>org.apache.maven.plugins</groupId> - <artifactId>maven-jar-plugin</artifactId> - </plugin> - </plugins> - <resources> - <resource> - <directory>src/main/resources</directory> - </resource> - </resources> - <finalName>${project.artifactId}-${project.version}</finalName> - </build> -</project> diff --git a/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlConnection.java b/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlConnection.java deleted file mode 100644 index 383a0f1380..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlConnection.java +++ /dev/null @@ -1,210 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service; - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; -import com.webank.wedatasphere.linkis.metadatamanager.common.domain.MetaColumnInfo; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.Closeable; -import java.io.IOException; -import java.sql.*; -import java.util.ArrayList; -import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; - -/** - * @author davidhua - * 2020/02/14 - */ -public class SqlConnection implements Closeable { - - private static final Logger LOG = LoggerFactory.getLogger(SqlConnection.class); - - private static final CommonVars<String> SQL_DRIVER_CLASS = - CommonVars.apply("wds.linkis.server.mdm.service.sql.driver", "com.mysql.jdbc.Driver"); - - private static final CommonVars<String> SQL_CONNECT_URL = - CommonVars.apply("wds.linkis.server.mdm.service.sql.url", "jdbc:mysql://%s:%s/%s"); - - private Connection conn; - - private ConnectMessage connectMessage; - - public SqlConnection(String host, Integer port, - String username, String password, - Map<String, Object> extraParams ) throws ClassNotFoundException, SQLException { - connectMessage = new ConnectMessage(host, port, username, password, extraParams); - conn = getDBConnection(connectMessage, ""); - //Try to create statement - Statement statement = conn.createStatement(); - statement.close(); - } - - public List<String> getAllDatabases() throws SQLException { - java.util.List<java.lang.String> dataBaseName = new ArrayList<>(); - Statement stmt = null; - ResultSet rs = null; - try{ - stmt = conn.createStatement(); - rs = stmt.executeQuery("SHOW DATABASES"); - while (rs.next()){ - dataBaseName.add(rs.getString(1)); - } - } finally { - closeResource(null, stmt, rs); - } - return dataBaseName; - } - - public List<String> getAllTables(String database) throws SQLException { - List<String> tableNames = new ArrayList<>(); - Statement stmt = null; - ResultSet rs = null; - try { - stmt = conn.createStatement(); - rs = stmt.executeQuery("SHOW TABLES FROM `" + database + "`"); - while (rs.next()) { - tableNames.add(rs.getString(1)); - } - return tableNames; - } finally{ - closeResource(null, stmt, rs); - } - } - - public List<MetaColumnInfo> getColumns(String database, String table) throws SQLException, ClassNotFoundException { - List<MetaColumnInfo> columns = new ArrayList<>(); - String columnSql = "SELECT * FROM `" + database +"`.`" + table + "` WHERE 1 = 2"; - PreparedStatement ps = null; - ResultSet rs = null; - ResultSetMetaData meta = null; - try { - List<String> primaryKeys = getPrimaryKeys(getDBConnection(connectMessage, database), table); - ps = conn.prepareStatement(columnSql); - rs = ps.executeQuery(); - meta = rs.getMetaData(); - int columnCount = meta.getColumnCount(); - for (int i = 1; i < columnCount + 1; i++) { - MetaColumnInfo info = new MetaColumnInfo(); - info.setIndex(i); - info.setName(meta.getColumnName(i)); - info.setType(meta.getColumnTypeName(i)); - if(primaryKeys.contains(meta.getColumnName(i))){ - info.setPrimaryKey(true); - } - columns.add(info); - } - }finally { - closeResource(null, ps, rs); - } - return columns; - } - - /** - * Get primary keys - * @param connection connection - * @param table table name - * @return - * @throws SQLException - */ - private List<String> getPrimaryKeys(Connection connection, String table) throws SQLException { - ResultSet rs = null; - List<String> primaryKeys = new ArrayList<>(); - try { - DatabaseMetaData dbMeta = connection.getMetaData(); - rs = dbMeta.getPrimaryKeys(null, null, table); - while(rs.next()){ - primaryKeys.add(rs.getString("column_name")); - } - return primaryKeys; - }finally{ - if(null != rs){ - closeResource(connection, null, rs); - } - } - } - - /** - * close database resource - * @param connection connection - * @param statement statement - * @param resultSet result set - */ - private void closeResource(Connection connection, Statement statement, ResultSet resultSet){ - try { - if(null != resultSet && !resultSet.isClosed()) { - resultSet.close(); - } - if(null != statement && !statement.isClosed()){ - statement.close(); - } - if(null != connection && !connection.isClosed()){ - connection.close(); - } - }catch (SQLException e){ - LOG.warn("Fail to release resource [" + e.getMessage() +"]", e); - } - } - - @Override - public void close() throws IOException { - closeResource(conn, null, null); - } - - /** - * @param connectMessage - * @param database - * @return - * @throws ClassNotFoundException - */ - private Connection getDBConnection(ConnectMessage connectMessage, String database) throws ClassNotFoundException, SQLException { - String extraParamString = connectMessage.extraParams.entrySet() - .stream().map(e -> String.join("=", e.getKey(), String.valueOf(e.getValue()))) - .collect(Collectors.joining("&")); - Class.forName(SQL_DRIVER_CLASS.getValue()); - String url = String.format(SQL_CONNECT_URL.getValue(), connectMessage.host, connectMessage.port, database); - if(!connectMessage.extraParams.isEmpty()) { - url += "?" + extraParamString; - } - return DriverManager.getConnection(url, connectMessage.username, connectMessage.password); - } - - /** - * Connect message - */ - private static class ConnectMessage{ - private String host; - - private Integer port; - - private String username; - - private String password; - - private Map<String, Object> extraParams; - - public ConnectMessage(String host, Integer port, - String username, String password, - Map<String, Object> extraParams){ - this.host = host; - this.port = port; - this.username = username; - this.password = password; - this.extraParams = extraParams; - } - } -} diff --git a/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlMetaService.java b/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlMetaService.java deleted file mode 100644 index 744ee4a051..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlMetaService.java +++ /dev/null @@ -1,79 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service; - -import com.webank.wedatasphere.linkis.metadatamanager.common.Json; -import com.webank.wedatasphere.linkis.metadatamanager.common.domain.MetaColumnInfo; -import com.webank.wedatasphere.linkis.metadatamanager.common.service.AbstractMetaService; -import com.webank.wedatasphere.linkis.metadatamanager.common.service.MetadataConnection; -import org.springframework.stereotype.Component; - -import java.sql.SQLException; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - -/** - * @author davidhua - * 2020/02/14 - */ -@Component -public class SqlMetaService extends AbstractMetaService<SqlConnection> { - @Override - public MetadataConnection<SqlConnection> getConnection(String operator, Map<String, Object> params) throws Exception { - String host = String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_HOST.getValue(), "")); - //After deserialize, Integer will be Double, Why? - Integer port = (Double.valueOf(String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_PORT.getValue(), 0)))).intValue(); - String username = String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_USERNAME.getValue(), "")); - String password = String.valueOf(params.getOrDefault(SqlParamsMapper.PARAM_SQL_PASSWORD.getValue(), "")); - Map<String, Object> extraParams = new HashMap<>(); - Object sqlParamObj = params.get(SqlParamsMapper.PARAM_SQL_EXTRA_PARAMS.getValue()); - if(null != sqlParamObj){ - if(!(sqlParamObj instanceof Map)){ - extraParams = Json.fromJson(String.valueOf(sqlParamObj), Map.class, String.class, Object.class); - }else{ - extraParams = (Map<String, Object>)sqlParamObj; - } - } - assert extraParams != null; - return new MetadataConnection<>(new SqlConnection(host, port, username, password, extraParams)); - } - - @Override - public List<String> queryDatabases(SqlConnection connection) { - try { - return connection.getAllDatabases(); - } catch (SQLException e) { - throw new RuntimeException("Fail to get Sql databases(获取数据库列表失败)", e); - } - } - - @Override - public List<String> queryTables(SqlConnection connection, String database) { - try { - return connection.getAllTables(database); - } catch (SQLException e) { - throw new RuntimeException("Fail to get Sql tables(获取表列表失败)", e); - } - } - - @Override - public List<MetaColumnInfo> queryColumns(SqlConnection connection, String database, String table) { - try { - return connection.getColumns(database, table); - } catch (SQLException | ClassNotFoundException e) { - throw new RuntimeException("Fail to get Sql columns(获取字段列表失败)", e); - } - } -} diff --git a/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlParamsMapper.java b/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlParamsMapper.java deleted file mode 100644 index c41285e4e0..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/java/com/webank/wedatasphere/linkis/metadatamanager/service/SqlParamsMapper.java +++ /dev/null @@ -1,38 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service; - -import com.webank.wedatasphere.linkis.common.conf.CommonVars; - -/** - * @author davidhua - * 2020/02/14 - */ -public class SqlParamsMapper { - - public static final CommonVars<String> PARAM_SQL_HOST = - CommonVars.apply("wds.linkis.server.mdm.service.sql.host", "host"); - - public static final CommonVars<String> PARAM_SQL_PORT = - CommonVars.apply("wds.linkis.server.mdm.service.sql.port", "port"); - - public static final CommonVars<String> PARAM_SQL_USERNAME = - CommonVars.apply("wds.linkis.server.mdm.service.sql.username", "username"); - - public static final CommonVars<String> PARAM_SQL_PASSWORD = - CommonVars.apply("wds.linkis.server.mdm.service.sql.password", "password"); - - public static final CommonVars<String> PARAM_SQL_EXTRA_PARAMS = - CommonVars.apply("wds.linkis.server.mdm.service.sql.params", "params"); -} diff --git a/datasource/metadatamanager/mysql/service/src/main/resources/application.yml b/datasource/metadatamanager/mysql/service/src/main/resources/application.yml deleted file mode 100644 index bb31831572..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/resources/application.yml +++ /dev/null @@ -1,23 +0,0 @@ -server: - port: 8294 -spring: - application: - name: mdm-service-mysql - -eureka: - client: - serviceUrl: - defaultZone: http://${ip}:${port}/eureka/ - instance: - metadata-map: - test: wedatasphere - -management: - endpoints: - web: - exposure: - include: refresh,info -logging: - config: classpath:log4j2.xml - - diff --git a/datasource/metadatamanager/mysql/service/src/main/resources/linkis.properties b/datasource/metadatamanager/mysql/service/src/main/resources/linkis.properties deleted file mode 100644 index f83c10d281..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/resources/linkis.properties +++ /dev/null @@ -1,21 +0,0 @@ -# -# Copyright 2019 WeBank -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -wds.linkis.server.mybatis.mapperLocations= -wds.linkis.server.mybatis.typeAliasesPackage= -wds.linkis.server.mybatis.BasePackage= -wds.linkis.server.restful.scan.packages= - -#sit -wds.linkis.server.version=v1 - diff --git a/datasource/metadatamanager/mysql/service/src/main/resources/log4j.properties b/datasource/metadatamanager/mysql/service/src/main/resources/log4j.properties deleted file mode 100644 index d5ee44b86b..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/resources/log4j.properties +++ /dev/null @@ -1,33 +0,0 @@ -# -# Copyright 2019 WeBank -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO \ No newline at end of file diff --git a/datasource/metadatamanager/mysql/service/src/main/resources/log4j2.xml b/datasource/metadatamanager/mysql/service/src/main/resources/log4j2.xml deleted file mode 100644 index 1c68190669..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/resources/log4j2.xml +++ /dev/null @@ -1,35 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - ~ Copyright 2019 WeBank - ~ Licensed under the Apache License, Version 2.0 (the "License"); - ~ you may not use this file except in compliance with the License. - ~ You may obtain a copy of the License at - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> - -<configuration status="error" monitorInterval="30"> - <appenders> - <Console name="Console" target="SYSTEM_OUT"> - <ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/> - <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/> - </Console> - <RollingFile name="RollingFile" fileName="logs/linkis.log" - filePattern="logs/$${date:yyyy-MM}/linkis-log-%d{yyyy-MM-dd}-%i.log"> - <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5level] [%-40t] %c{1.} (%L) [%M] - %msg%xEx%n"/> - <SizeBasedTriggeringPolicy size="100MB"/> - <DefaultRolloverStrategy max="20"/> - </RollingFile> - </appenders> - <loggers> - <root level="INFO"> - <appender-ref ref="RollingFile"/> - <appender-ref ref="Console"/> - </root> - </loggers> -</configuration> - diff --git a/datasource/metadatamanager/mysql/service/src/main/scala/com/webank/wedatasphere/linkis/metadatamanager/service/receiver/SqlReceiver.scala b/datasource/metadatamanager/mysql/service/src/main/scala/com/webank/wedatasphere/linkis/metadatamanager/service/receiver/SqlReceiver.scala deleted file mode 100644 index 11f24a9ee9..0000000000 --- a/datasource/metadatamanager/mysql/service/src/main/scala/com/webank/wedatasphere/linkis/metadatamanager/service/receiver/SqlReceiver.scala +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Copyright 2019 WeBank - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * http://www.apache.org/licenses/LICENSE-2.0 - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.webank.wedatasphere.linkis.metadatamanager.service.receiver - -import com.webank.wedatasphere.linkis.DataWorkCloudApplication -import com.webank.wedatasphere.linkis.metadatamanager.common.receiver.BaseMetaReceiver -import com.webank.wedatasphere.linkis.metadatamanager.common.service.MetadataService -import javax.annotation.PostConstruct -import org.springframework.stereotype.Component - -@Component -class SqlReceiver extends BaseMetaReceiver{ - @PostConstruct - def init(): Unit = { - metadataService = DataWorkCloudApplication.getApplicationContext.getBean(classOf[MetadataService]) - } -} diff --git a/datasourcemanager/server/src/main/resources/log4j.properties b/datasourcemanager/server/src/main/resources/log4j.properties deleted file mode 100644 index d5ee44b86b..0000000000 --- a/datasourcemanager/server/src/main/resources/log4j.properties +++ /dev/null @@ -1,33 +0,0 @@ -# -# Copyright 2019 WeBank -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# http://www.apache.org/licenses/LICENSE-2.0 -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -### set log levels ### - -log4j.rootCategory=INFO,console - -log4j.appender.console=org.apache.log4j.ConsoleAppender -log4j.appender.console.Threshold=INFO -log4j.appender.console.layout=org.apache.log4j.PatternLayout -#log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n -log4j.appender.console.layout.ConversionPattern= %d{ISO8601} %-5p (%t) %p %c{1} - %m%n - - -log4j.appender.com.webank.bdp.ide.core=org.apache.log4j.DailyRollingFileAppender -log4j.appender.com.webank.bdp.ide.core.Threshold=INFO -log4j.additivity.com.webank.bdp.ide.core=false -log4j.appender.com.webank.bdp.ide.core.layout=org.apache.log4j.PatternLayout -log4j.appender.com.webank.bdp.ide.core.Append=true -log4j.appender.com.webank.bdp.ide.core.File=logs/linkis.log -log4j.appender.com.webank.bdp.ide.core.layout.ConversionPattern= %d{ISO8601} %-5p (%t) [%F:%M(%L)] - %m%n - -log4j.logger.org.springframework=INFO \ No newline at end of file diff --git a/datasourcemanager/server/src/main/resources/log4j2.xml b/datasourcemanager/server/src/main/resources/log4j2.xml deleted file mode 100644 index 1c68190669..0000000000 --- a/datasourcemanager/server/src/main/resources/log4j2.xml +++ /dev/null @@ -1,35 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - ~ Copyright 2019 WeBank - ~ Licensed under the Apache License, Version 2.0 (the "License"); - ~ you may not use this file except in compliance with the License. - ~ You may obtain a copy of the License at - ~ http://www.apache.org/licenses/LICENSE-2.0 - ~ Unless required by applicable law or agreed to in writing, software - ~ distributed under the License is distributed on an "AS IS" BASIS, - ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - ~ See the License for the specific language governing permissions and - ~ limitations under the License. - --> - -<configuration status="error" monitorInterval="30"> - <appenders> - <Console name="Console" target="SYSTEM_OUT"> - <ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/> - <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5level [%t] %logger{36} %L %M - %msg%xEx%n"/> - </Console> - <RollingFile name="RollingFile" fileName="logs/linkis.log" - filePattern="logs/$${date:yyyy-MM}/linkis-log-%d{yyyy-MM-dd}-%i.log"> - <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss.SSS} [%-5level] [%-40t] %c{1.} (%L) [%M] - %msg%xEx%n"/> - <SizeBasedTriggeringPolicy size="100MB"/> - <DefaultRolloverStrategy max="20"/> - </RollingFile> - </appenders> - <loggers> - <root level="INFO"> - <appender-ref ref="RollingFile"/> - <appender-ref ref="Console"/> - </root> - </loggers> -</configuration> - diff --git a/pom.xml b/pom.xml index 79b182cef7..db47d54209 100644 --- a/pom.xml +++ b/pom.xml @@ -101,6 +101,14 @@ <module>ujes/definedEngines/hive/enginemanager</module> <module>ujes/definedEngines/hive/entrance</module> + <module>datasource/datasourcemanager/common</module> + <module>datasource/datasourcemanager/server</module> + <module>datasource/metadatamanager/common</module> + <module>datasource/metadatamanager/server</module> + <module>datasource/metadatamanager/service/elasticsearch</module> + <module>datasource/metadatamanager/service/hive</module> + <module>datasource/metadatamanager/service/mysql</module> + <module>ujes/definedEngines/python/engine</module> <module>ujes/definedEngines/python/enginemanager</module> <module>ujes/definedEngines/python/entrance</module> diff --git a/ujes/definedEngines/jdbc/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/executer/JDBCEngineExecutor.scala b/ujes/definedEngines/jdbc/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/executer/JDBCEngineExecutor.scala index c53b2691a7..cdb089cdef 100644 --- a/ujes/definedEngines/jdbc/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/executer/JDBCEngineExecutor.scala +++ b/ujes/definedEngines/jdbc/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/executer/JDBCEngineExecutor.scala @@ -15,7 +15,7 @@ */ package com.webank.wedatasphere.linkis.entrance.executer -import java.sql.{SQLFeatureNotSupportedException, Statement} +import java.sql.{Connection, SQLFeatureNotSupportedException, Statement} import java.util import com.webank.wedatasphere.linkis.common.io.FsPath @@ -43,6 +43,7 @@ class JDBCEngineExecutor(outputPrintLimit: Int, properties: util.HashMap[String, private val LOG = LoggerFactory.getLogger(getClass) private val connectionManager = ConnectionManager.getInstance() private var statement: Statement = null + private var connection:Connection = null private val name: String = Sender.getThisServiceInstance.getInstance private val persistEngine = new EntranceResultSetEngine() //execute line number，as alias and progress line @@ -53,7 +54,7 @@ class JDBCEngineExecutor(outputPrintLimit: Int, properties: util.HashMap[String, protected def executeLine(code: String, storePath: String, alias: String): ExecuteResponse = { val realCode = code.trim() LOG.info(s"jdbc client begins to run jdbc code:\n ${realCode.trim}") - val connection = connectionManager.getConnection(properties) + connection = connectionManager.getConnection(properties) statement = connection.createStatement() LOG.info(s"create statement is: $statement") val isResultSetAvailable = statement.execute(code) @@ -202,6 +203,11 @@ class JDBCEngineExecutor(outputPrintLimit: Int, properties: util.HashMap[String, totalCodeLineNumber=0 logger.error("JDBC query failed", t) return ErrorExecuteResponse("JDBC query failed", t) + }finally { + Utils.tryQuietly{ + statement.close() + connection.close() + } } } } diff --git a/ujes/entrance/src/main/java/com/webank/wedatasphere/linkis/entrance/parser/AbstractEntranceParser.java b/ujes/entrance/src/main/java/com/webank/wedatasphere/linkis/entrance/parser/AbstractEntranceParser.java index 634a48af0b..5b0d1a3ddf 100644 --- a/ujes/entrance/src/main/java/com/webank/wedatasphere/linkis/entrance/parser/AbstractEntranceParser.java +++ b/ujes/entrance/src/main/java/com/webank/wedatasphere/linkis/entrance/parser/AbstractEntranceParser.java @@ -75,6 +75,11 @@ public abstract class AbstractEntranceParser extends EntranceParser { && StringUtils.isNotEmpty(job.getErrorResponse().message())) { ((RequestPersistTask) task).setErrDesc(job.getErrorResponse().message()); } + //if job is successful, errCode and errDesc needs to be null + if (job.isSucceed()){ + ((RequestPersistTask) task).setErrCode(null); + ((RequestPersistTask) task).setErrDesc(null); + } }else{ logger.warn("not supported task type"); } diff --git a/ujes/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/interceptor/impl/CommentInterceptor.scala b/ujes/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/interceptor/impl/CommentInterceptor.scala index 0d9534b86c..0fa8c97cba 100644 --- a/ujes/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/interceptor/impl/CommentInterceptor.scala +++ b/ujes/entrance/src/main/scala/com/webank/wedatasphere/linkis/entrance/interceptor/impl/CommentInterceptor.scala @@ -47,6 +47,7 @@ class CommentInterceptor extends EntranceInterceptor { case "sql" | "hql" => requestPersistTask.setExecutionCode(SQLCommentHelper.dealComment(requestPersistTask.getExecutionCode)) case "python" | "py" => requestPersistTask.setExecutionCode(PythonCommentHelper.dealComment(requestPersistTask.getExecutionCode)) case "scala" | "java" => requestPersistTask.setExecutionCode(ScalaCommentHelper.dealComment(requestPersistTask.getExecutionCode)) + case "sh" | "shell" => case _ => requestPersistTask.setExecutionCode(SQLCommentHelper.dealComment(requestPersistTask.getExecutionCode)) } requestPersistTask --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
