log SPI +1
在 2020-11-04 10:07:58，"JUN GAO" <[email protected]> 写道： >log SPI +1 > >But I think this is a low priority task, we should give priority to >complete the registration center SPI and task SPI. > > > >CalvinKirs <[email protected]> 于2020年11月3日周二 下午11:04写道： > >> Thank you very much for your answer, I misunderstood what you meant. >> >> >> log SPI+1 >> >> >> Best wishes！ >> CalvinKirs >> >> >> On 11/3/2020 22:55，leon bao<[email protected]> wrote： >> Maybe it didn't express my meaning well. >> Yes, DolphinScheduler doesn't need to do this storage. We just need to do >> the log plug-in function to support users to implement their own log >> storage mode, just like we do alert plug-ins. >> >> >> 可能是没有表达好的我的意思。 >> 是的，DolphinScheduler没必要去做这个存储，我们只需要做日志插件功能，支持用户实现自己的日志存储方式，就像我们做告警插件一样。 >> >> >> boyi <[email protected]> 于2020年11月3日周二 下午10:37写道： >> >> hi: >> >> >> Agree with Calvin KIRs >> >> >> >> >> >> >> Generally, there will be a unified log collection for such files, such as >> ELK. Especially when deployed in the docker environment, we can consider >> focusing on more important things >> >> >> -------------------------------------- >> BoYi ZhangE-mail : [email protected] >> On 11/3/2020 22:23，CalvinKirs<[email protected]> wrote： >> I understand what you mean, but what I want to express is whether it would >> be better for users to do this part of the work. Generally, companies have >> their own log collection system. If we were to do this part of the storage, >> the workload would be great, and the benefits would not be great. >> >> >> >> >> 我明白您的意思，但我想表达的是，这部分工作交给用户来做会不会更好一点，一般企业都会有自己的日志收集系统。如果我们来做这部分存储的话工作量会很大，并且收益并不是很可观。 >> >> >> Best wishes！ >> CalvinKirs >> >> >> On 11/3/2020 22:09，leon bao<[email protected]> wrote： >> @CalvinKirs >> >> about log spi , now have a requirement in scalable >> services(master/worker), this kind of application scenario requires that >> the task log cannot be stored in the worker / master, but needs to be >> stored in a third-party place, which maybe database or other >> storage. Therefore, if the dolphin scheduler can provide this plug-in >> function, users can read and write logs according to their own needs. >> >> >> 关于日志插件，我们现在有一个需求是可伸缩的服务(master/worker)，这种应用场景就要求任务的日志不能存在某一个worker/master上， >> >> >> 而是需要存在一个第三方的地方，可能是数据库或者其他存储。所以如果DolphinScheduler能提供这个插件功能，用户就可以根据自己需求来实现日志的读写. >> >> CalvinKirs <[email protected]> 于2020年11月3日周二 下午6:40写道： >> >> >> >> Great planning! >> But I have a little question, what are the specific requirements of the >> log SPI? I am not very clear at present, are we only implementing SPI for >> data storage? If this is the case, is it necessary? I think this user can >> use logagent (or other technologies) for related implementations. Different >> users have different needs. Some users may also involve aggregation, >> calculation, and even different magnitudes, and may use additional >> components. Therefore, if we store the original data in this piece, a lot >> of redundant data may be generated. >> >> >> 非常棒的规划！ >> >> >> 但是我有一点疑问，日志SPI这块的具体需求是什么？我目前不是很明确，我们是只对数据存储做SPI实现吗？如果是这样的话，是否有必要呢？我认为这块用户可以自己使用logagent（或者其他技术） >> >> >> 来进行相关的实现，不同用户的需求不同，有的用户可能还牵扯到聚合、计算，甚至量级不同，还有可能使用额外的组件。因此，这块如果我们来对原始数据做存储的话可能会产生很多冗余数据。 >> >> >> Best wishes！ >> CalvinKirs >> >> >> On 11/3/2020 11:47，leon bao<[email protected]> wrote： >> Hello Everyone: >> >> DS has good horizontal scalability with its non central design >> architecture, which attracts many developers. With more and more users, the >> demand for scheduling is becoming more and more complex. >> At the same time, the functional design of DS is required to be more >> scalable，for example: the plug-in function of alarm mode. >> So we can discuss what parts of plug-ins DS can do at present. We can >> reconstruct DolphinScheduler according to the results of this discussion. >> At present, there are several parts of demand: >> >> - alert model: >> refer to: >> https://github.com/apache/incubator-dolphinscheduler/issues/3049 >> >> - task plugin: >> refer to: >> https://github.com/apache/incubator-dolphinscheduler/issues/2869 >> >> - register center: >> refer to: >> >> >> >> >> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E >> https://github.com/apache/incubator-dolphinscheduler/issues/3961 >> >> - log model: >> The current log is recorded by writing local files of the server. >> Can we make this plug-in type, which can facilitate users to extend the >> log reading and writing types, such as writing to the database or other >> third-party systems. >> >> - global task queue >> At present, tasks are stored in the memory queue of the master, which >> results in the priority of a task can only work within the scope of a >> master. >> In order to make the priority of a task effective globally, we need a >> global queue to make the global priority work. >> (in version 1.2, we used zookeeper as the global queue, which was removed >> because of the delay of ZK operation) >> >> Implementation details can be discussed within each topic. Here, we only >> discuss the requirements. >> Very appreciate you can put forward more opinions. >> >> >> >> >> ================================================================================================================================================================== >> >> >> >> >> DS目前以无中心的设计架构具备了很好的横向扩展性，这个特性吸引了很多的开发者。随着DS用户越来越多，对调度的需求越来越复杂，同时也要求DS在功能设计上要更具有可扩展性 >> 比如告警方式的插件功能，所以在这里大家可以讨论目前DS可以做哪些部分的插件，后续我们可以根据这个讨论结果，来对DS进行插件方面的重构。 >> 目前已经有需求的几个部分： >> >> - 告警插件(running) >> 相关讨论： >> https://github.com/apache/incubator-dolphinscheduler/issues/3049 >> >> - 任务插件 >> https://github.com/apache/incubator-dolphinscheduler/issues/2869 >> >> - 注册中心 >> 相关讨论： >> >> >> >> >> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E >> https://github.com/apache/incubator-dolphinscheduler/issues/3961 >> >> - 日志插件 >> 目前的日志是通过写服务器本地文件的形式记录的，是不是可以把这个做成插件类型，方便用户扩展日志读写类型，比如写到数据库或者其他第三方系统中。 >> >> - 全局队列插件 >> >> >> >> >> 目前任务是被存储在master的内存队列，这就导致了任务的优先级只能在一定范围内起作用，为了让任务的优先级在全局有效，我们需要一种全局队列来让全局优先级起作用。(比如1.2版本我们使用的zookeeper作为全局队列,后面因为zk操作的延时性我们去掉了这个)。 >> >> 实现细节可以在每个话题内部进行讨论，在这里我们只讨论需求，希望大家可以提出更多意见。 >> >> -- >> DolphinScheduler(Incubator) PPMC >> BaoLiang 鲍亮 >> [email protected] >> >> >> >> -- >> DolphinScheduler(Incubator) PPMC >> BaoLiang 鲍亮 >> [email protected] >> >> >> >> -- >> DolphinScheduler(Incubator) PPMC >> BaoLiang 鲍亮 >> [email protected] >> > > >-- > >DolphinScheduler(Incubator) PPMC >Jun Gao 高俊 >[email protected]
