+1 In fact, before improving spi, I have seen some issue and pr about this part of the content. Contributors want to integrate other storage methods, but the related license may inconformity??Therefore, I support your point of view...
| | Kirs | | [email protected] | Signature is customized by Netease Mail Master ??2020??11??10?? 17:36??zixi0825 ?????? File resource SPI , the current DS file resource storage supports HDFS, S3 and NONE, making plug-in allows users to customize storage methods to meet various storage needs And the task spi should be the highest priority =============================================================================================================== ????????????????????DS????????????????????HDFS??S3 and NONE?????????????????????????????????????????????????????????? Task SPI ?????????????????????? ------------------ ???????? ------------------ ??????: "dev" <[email protected]>; ????????: 2020??11??4??(??????) ????10:20 ??????: "dev"<[email protected]>; ????: Re:Re: [DISCUSS]Which part of DolphinScheduler can be more scalable? log SPI +1 ?? 2020-11-04 10:07:58??"JUN GAO" <[email protected]> ?????? >log SPI +1 > >But I think this is a low priority task, we should give priority to >complete the registration center SPI and task SPI. > > > >CalvinKirs <[email protected]> ??2020??11??3?????? ????11:04?????? > >> Thank you very much for your answer, I misunderstood what you meant. >> >> >> log SPI+1 >> >> >> Best wishes?? >> CalvinKirs >> >> >> On 11/3/2020 22:55??leon bao<[email protected]> wrote?? >> Maybe it didn't express my meaning well. >> Yes, DolphinScheduler doesn't need to do this storage. We just need to do >> the log plug-in function to support users to implement their own log >> storage mode, just like we do alert plug-ins. >> >> >> ???????????????????????????? >> ??????DolphinScheduler?????????????????????????????????????????????????????????????????????????????????????????????????????? >> >> >> boyi <[email protected]> ??2020??11??3?????? ????10:37?????? >> >> hi: >> >> >> Agree with Calvin KIRs >> >> >> >> >> >> >> Generally, there will be a unified log collection for such files, such as >> ELK. Especially when deployed in the docker environment, we can consider >> focusing on more important things >> >> >> -------------------------------------- >> BoYi ZhangE-mail : [email protected] >> On 11/3/2020 22:23??CalvinKirs<[email protected]> wrote?? >> I understand what you mean, but what I want to express is whether it would >> be better for users to do this part of the work. Generally, companies have >> their own log collection system. If we were to do this part of the storage, >> the workload would be great, and the benefits would not be great. >> >> >> >> >> ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> >> >> Best wishes?? >> CalvinKirs >> >> >> On 11/3/2020 22:09??leon bao<[email protected]> wrote?? >> @CalvinKirs >> >> about log spi , now have a requirement in scalable >> services(master/worker), this kind of application scenario requires that >> the task log cannot be stored in the worker / master, but needs to be >> stored in a third-party place, which maybe database or other >> storage. Therefore, if the dolphin scheduler can provide this plug-in >> function, users can read and write logs according to their own needs. >> >> >> ??????????????????????????????????????????????(master/worker)????????????????????????????????????????????worker/master???? >> >> >> ????????????????????????????????????????????????????????????????DolphinScheduler??????????????????????????????????????????????????????????. >> >> CalvinKirs <[email protected]> ??2020??11??3?????? ????6:40?????? >> >> >> >> Great planning! >> But I have a little question, what are the specific requirements of the >> log SPI? I am not very clear at present, are we only implementing SPI for >> data storage? If this is the case, is it necessary? I think this user can >> use logagent (or other technologies) for related implementations. Different >> users have different needs. Some users may also involve aggregation, >> calculation, and even different magnitudes, and may use additional >> components. Therefore, if we store the original data in this piece, a lot >> of redundant data may be generated. >> >> >> ?????????????? >> >> >> ??????????????????????SPI????????????????????????????????????????????????????????????SPI????????????????????????????????????????????????????????????????logagent???????????????? >> >> >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> >> >> Best wishes?? >> CalvinKirs >> >> >> On 11/3/2020 11:47??leon bao<[email protected]> wrote?? >> Hello Everyone: >> >> DS has good horizontal scalability with its non central design >> architecture, which attracts many developers. With more and more users, the >> demand for scheduling is becoming more and more complex. >> At the same time, the functional design of DS is required to be more >> scalable??for example: the plug-in function of alarm mode. >> So we can discuss what parts of plug-ins DS can do at present. We can >> reconstruct DolphinScheduler according to the results of this discussion. >> At present, there are several parts of demand: >> >> - alert model: >> refer to: >> https://github.com/apache/incubator-dolphinscheduler/issues/3049 >> >> - task plugin: >> refer to: >> https://github.com/apache/incubator-dolphinscheduler/issues/2869 >> >> - register center: >> refer to: >> >> >> >> >> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E >> https://github.com/apache/incubator-dolphinscheduler/issues/3961 >> >> - log model: >> The current log is recorded by writing local files of the server. >> Can we make this plug-in type, which can facilitate users to extend the >> log reading and writing types, such as writing to the database or other >> third-party systems. >> >> - global task queue >> At present, tasks are stored in the memory queue of the master, which >> results in the priority of a task can only work within the scope of a >> master. >> In order to make the priority of a task effective globally, we need a >> global queue to make the global priority work. >> (in version 1.2, we used zookeeper as the global queue, which was removed >> because of the delay of ZK operation) >> >> Implementation details can be discussed within each topic. Here, we only >> discuss the requirements. >> Very appreciate you can put forward more opinions. >> >> >> >> >> ================================================================================================================================================================== >> >> >> >> >> DS??????????????????????????????????????????????????????????????????????????????DS????????????????????????????????????????????????DS???????????????????????????? >> ??????????????????????????????????????????????????DS????????????????????????????????????????????????????????DS???????????????????? >> ?????????????????????????? >> >> - ????????(running) >> ?????????? >> https://github.com/apache/incubator-dolphinscheduler/issues/3049 >> >> - ???????? >> https://github.com/apache/incubator-dolphinscheduler/issues/2869 >> >> - ???????? >> ?????????? >> >> >> >> >> https://lists.apache.org/thread.html/r755a57e3b859563de2dddf8aa2f336fcf28934e7bbb2c3f97fe5fe3d%40%3Cdev.dolphinscheduler.apache.org%3E >> https://github.com/apache/incubator-dolphinscheduler/issues/3961 >> >> - ???????? >> ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> >> - ???????????? >> >> >> >> >> ??????????????????master??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????(????1.2??????????????zookeeper????????????,????????zk??????????????????????????)?? >> >> ???????????????????????????????????????????????????????????????????????????????????? >> >> -- >> DolphinScheduler(Incubator) PPMC >> BaoLiang ???? >> [email protected] >> >> >> >> -- >> DolphinScheduler(Incubator) PPMC >> BaoLiang ???? >> [email protected] >> >> >> >> -- >> DolphinScheduler(Incubator) PPMC >> BaoLiang ???? >> [email protected] >> > > >-- > >DolphinScheduler(Incubator) PPMC >Jun Gao ???? >[email protected]
