hi, This is not a formal meeting, just to understand what the developers think about DolphinScheduler, so I didn’t inform you in advance when the meeting was held in the email list. I think we could hold online meeting to discuss the topic about "how to refactor" this week
Best Regards --------------- DolphinScheduler(Incubator) PPMC Lidong Dai 代立冬 [email protected] --------------- On Mon, Nov 23, 2020 at 10:01 AM Sheng Wu <[email protected]> wrote: > Hi Calvin > > I think you misunderstood the point about Furkan's asking. He was asking > about how do you set up the meeting which should be an open way. > > Sheng Wu 吴晟 > Twitter, wusheng1108 > > > CalvinKirs <[email protected]> 于2020年11月23日周一 上午9:47写道： > > > Hi Furkan, welcome very much. This meeting is a Chinese video conference. > > We will send meeting invitations in the Chinese contributor exchange > group. > > The relevant design conclusions will be reflected in the dev mailing > list, > > and you are very welcome to discuss with you in the mail. > > > > > > In addition, thank you very much for your reminder. :in the later days > > later online meetings, we will send meeting invitations in advance on the > > mailing list, and you can also communicate with us during the meeting. > > > > > > Best wishes！ > > CalvinKirs > > > > > > On 11/22/2020 23:05，Furkan KAMACI<[email protected]> wrote： > > Hi BoYi, > > > > May I learn at where you decided to do this meeting (slack, mail list > > etc.)? > > > > Kind Regards, > > Furkan KAMACI > > > > On Thu, Nov 19, 2020 at 10:21 AM wu shaoj <[email protected]> wrote: > > > > Cool, excellent meeting > > > > > > From: boyi <[email protected]> > > Date: Thursday, November 19, 2020 at 15:18 > > To: [email protected] <[email protected]> > > Subject: [Online Meeting Invitation]DolphinScheduler refactor > > meeting(Second) > > [Online Meeting Invitation]DolphinScheduler refactor meeting(Second) > > > > > > Hi, DolphinScheduler Community: > > > > > > > > > > We discussed the DolphinScheduler reconstruction workflow definition > > storage structure (split JSON data) at 2020-11-17 19:00 Beijing time. A > > total of 10+ partners participated in this meeting. The discussion > > information of the meeting is as follows: > > > > > > 1: Currently, we provide an idea of splitting the workflow definition > > table (t_ds_process_definition) into two tables. The data structure > > information is as follows: > > > > > > t_ds_process_definition[subject table] and t_ds_process_definition_task > > [task detail table] > > > > > > | t_ds_process_definition | > > | name | type | describe | > > | id | int(11) | id | > > | name | varchar(255) | process definition name | > > | version | int(11) | process definition version | > > | release_state | tinyint(4) | 0 is not online, 1 is online | > > | project_id | int(11) | project id | > > | user_id | int(11) | user id | > > | description | text | description | > > | global_params | text | global params | > > | flag | tinyint(4) | 0 is not available, 1 is available | > > | receivers | text | addressee | > > | receivers_cc | text | CC person | > > | create_time | datetime | create time | > > | timeout | int(11) | time out | > > | tenant_id | int(11) | tenant id | > > | update_time | datetime | update time | > > | modify_by | varchar(36) | modify by user name | > > > > > > | t_ds_process_definition_task | > > | name | type | describe | > > | id | int(11) | task id | > > | name | varchar(255) | task name | > > | type | varchar(64) | type [SHLL,PYTHON,DATAX,SPARK 等等 ] | > > | process_definition_id | int(11) | process definition id | > > | params | longtext | custom parameters [JSON ] | > > | description | text | description | > > | runFlag | tinyint(4) | operation identification | > > | conditionResult | longtext | conditional branch [JSON ] | > > | dependence | longtext | task dependency [JSON ] | > > | maxRetryTimes | tinyint(4) | max retry times | > > | retryInterval | tinyint(4) | retry interval | > > | timeout | varchar(128) | time out [JSON ] | > > | taskInstancePriority | varchar(16) | task priority | > > | workerGroup | varchar(64) | worker group name | > > | preTasks | varchar(128) | pre task | > > | locations | text | dag location | > > | connects | text | dag connect | > > | resource | varchar(255) | resouce mark | > > | datasource | varchar(255) | datasource mark | > > > > > > > > > > 2: Consider whether you need a third table to store the dependencies > > between tasks, mainly for workflow dependent nodes, condition judgments, > > and task bloodlines for higher-level abstraction. Split into the third > > table. > > > > > > > > > > > > > > 3. Identify the issues to be discussed in the next meeting > > 3.1. Is the workflow definition table split into two tables or into > > three tables? > > 3.2. How to store data sources and resource files > > 3.3. How to store workflow task instances without affecting > > re-running, and to support existing functions such as editing workflow > > definitions. > > 3.4. Workflow definition version issue > > > > > > We are very grateful to the following friends for their discussions: > > dailidong, lgcareer, CalvinKirs, Rubik-W, leonbao, zixi0825, > JinyLeeChina, > > chenxingchun, BoYiZhang, etc. They provided a lot of effective > suggestions > > for this meeting. > > > > > > At the same time, the community also hopes that more people can > > participate. Thank you very much. > > > > > > Best wishes！ > > BoYiZhang > > > > > > > > > > > > > > ------------------------------------------------ > > > > > > hi，DolphinScheduler 社区： > > > > > > 我们在北京时间2020-11-17 > > 19:00针对DolphinScheduler重构工作流定义存储结构(拆大JSON)进行了讨论，共有10+位伙伴参与了本次会议，会议讨论信息如下： > > > > > > 1:目前提供一个将工作流定义表(t_ds_process_definition)拆分为两张表的思路,数据结构信息如下 : > > t_ds_process_definition[主体表] 和 t_ds_process_definition_task [任务详情表] > > > > > > > > > > > > > > 2:考虑是否需要第三张表存储任务之间的依赖关系, 主要针对工作流依赖节点,条件判断,任务血缘进行更高层级的抽象.拆分成第三张表. > > > > > > | t_ds_process_definition[主体表] | > > | 序号 | 字段 | 类型 | 描述 | > > | 1 | id | int(11) | 主键 | > > | 2 | name | varchar(255) | 流程定义名称 | > > | 3 | version | int(11) | 流程定义版本 | > > | 4 | release_state | tinyint(4) | 流程定义的发布状态：0 未上线 , 1已上线 | > > | 5 | project_id | int(11) | 项目id | > > | 6 | user_id | int(11) | 流程定义所属用户id | > > | 7 | description | text | 流程定义描述 | > > | 8 | global_params | text | 全局参数 | > > | 9 | flag | tinyint(4) | 流程是否可用：0 不可用，1 可用 | > > | 10 | receivers | text | 收件人 | > > | 11 | receivers_cc | text | 抄送人 | > > | 12 | create_time | datetime | 创建时间 | > > | 13 | timeout | int(11) | 超时时间 | > > | 14 | tenant_id | int(11) | 租户id | > > | 15 | update_time | datetime | 更新时间 | > > | 16 | modify_by | varchar(36) | 修改用户 | > > > > > > | t_ds_process_definition_task [任务详情表] | > > | 序号 | 参数名 | 类型 | 描述 | > > | 1 | id | int(11) | 任务id | > > | 2 | name | varchar(255) | 任务名称 | > > | 3 | type | varchar(64) | 类型 [SHLL,PYTHON,DATAX,SPARK 等等 ] | > > | 4 | process_definition_id | int(11) | 流程定义id | > > | 5 | params | longtext | 自定义参数 [ Json 格式 保存原有的params字段, > 自定义参数和资源文件参数是否拆出 > > ] | > > | 6 | description | text | 描述 | > > | 7 | runFlag | tinyint(4) | 运行标识 | > > | 8 | conditionResult | longtext | 条件分支 [JSON格式] | > > | 9 | dependence | longtext | 任务依赖 [JSON格式] | > > | 10 | maxRetryTimes | tinyint(4) | 最大重试次数 | > > | 11 | retryInterval | tinyint(4) | 重试间隔 | > > | 12 | timeout | varchar(128) | 超时控制策略 [JSON格式] | > > | 13 | taskInstancePriority | varchar(16) | 任务优先级 | > > | 14 | workerGroup | varchar(64) | Worker 分组名称 | > > | 15 | preTasks | varchar(128) | 前置任务 | > > | 16 | locations | text | 节点坐标信息 | > > | 17 | connects | text | 节点连线信息 | > > | 18 | resource | varchar(255) | 资源文件标识 , 以逗号分隔 | > > | 19 | datasource | varchar(255) | 数据源标识 , 以逗号分隔 | > > > > > > 3.明确下次会议待讨论问题 > > 3.1. 工作流定义表是拆成两张表还是拆成三张表? > > 3.2. 数据源和资源文件如何存储 > > 3.3. 工作流任务实例如何存储,不影响重跑 ,要支持编辑工作流定义等现有的功能. > > 3.4. 工作流定义版本问题 > > > > > > > > > > > > > 我们很感谢以下朋友的讨论：dailidong、lgcareer、CalvinKirs、Rubik-W、leonbao、zixi0825、JinyLeeChina、chenxingchun > > 、 BoYiZhang 等，他们对本次会议提供了很多有效的建议。 > > > > > > 同时社区也希望更多的人能够参与进来。非常感谢你们。 > > > > > > -------------------------------------- > > BoYi ZhangE-mail : [email protected] > > > > >
