This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new d5ee1624d3 Add Blog News (#836)
d5ee1624d3 is described below
commit d5ee1624d3bb21aaa906040fbdd66bffad476509
Author: lifeng <[email protected]>
AuthorDate: Mon Oct 31 20:36:50 2022 +0800
Add Blog News (#836)
* Add Blog News
* Update
How_did_Yili_explore_a_path_for_digital_transformation_based_on_DolphinScheduler.md
Co-authored-by: Jiajie Zhong <[email protected]>
---
blog/en-us/Apache_DolphinScheduler_2.0.7.md | 60 ++++++
.../How_can_more_people_benefit_from_big_data.md | 210 +++++++++++++++++++++
...tal_transformation_based_on_DolphinScheduler.md | 137 ++++++++++++++
site_config/blog.js | 21 +++
site_config/home.jsx | 15 +-
5 files changed, 435 insertions(+), 8 deletions(-)
diff --git a/blog/en-us/Apache_DolphinScheduler_2.0.7.md
b/blog/en-us/Apache_DolphinScheduler_2.0.7.md
new file mode 100644
index 0000000000..bbfb2e68af
--- /dev/null
+++ b/blog/en-us/Apache_DolphinScheduler_2.0.7.md
@@ -0,0 +1,60 @@
+---
+title:# Apache DolphinScheduler 2.0.7 released, fixes complement and fault
tolerance
+keywords: Apache,DolphinScheduler,scheduler,big
data,ETL,airflow,hadoop,orchestration,dataops,Meetup
+description: Recently, Apache DolphinScheduler version 2.0.7 was officially
released...
+---
+# Apache DolphinScheduler 2.0.7 released, fixes complement and fault tolerance
+
+
+
+Recently, Apache DolphinScheduler version 2.0.7 was officially released. The
new version fixes some complement and fault tolerance bugs and solves problems
such as inconsistent batch data.
+
+# Major bug fixes
+
+## №1 The problem of continued detection of dependent while the dependent node
contains tasks that are prohibited from running
+* In version 2.0.6, when a dependent node has a task that is forbidden to run,
the dependent node will continue to detect the task instance for unable to find
the task instance, since the forbidden task does not generate a task instance,
resulting in the post-task of the dependent node not being submitted. This
issue has been fixed in version 2.0.7.
+* PR: https://github.com/apache/dolphinscheduler/pull/10952
+
+## №2 The problem of +1 day to the complement time by default
+* In the previous version, when October 1, 2022 was selected as the complement
time, for the time parameter $[yyyy-MM-dd], the output date was 2022–10–02,
which is not the expected date. To solve this problem, the new version
subtracts one day from the complement date to correct the output date.
+* Corresponding PR: https://github.com/apache/dolphinscheduler/pull/12376
+## №3 The problem that the task cannot be resubmitted when it is fault-tolerant
+
+* In the previous version, if there are 2 Worker nodes, each Worker is running
10 tasks and 10 internally -queued tasks. When Worker 1 hangs up, the state
will turn to a fault-tolerant state because the status of all the running tasks
and the tasks waiting to be run needs to be fault-tolerant. However, due to a
bug, the fault-tolerant task will not be resubmitted, which will cause the
running task background to show that the running is completed, while the task
waiting to be run does not [...]
+* Corresponding to PR https://github.com/apache/dolphinscheduler/pull/12423
+
+# Bug fix
+
+* Fix the problem that the disk monitoring in the monitoring interface is not
displayed
+* Fixed the problem of transferring parameters for tasks that failed to restore
+* Fix the problem that the dependency node contains forbidden tasks and the
dependency continues to detect
+* Fix HTTP alert including content field in Post
+* Fix the problem that the complement time is +1 day by default
+* Fixed an issue where the retry time of failed tasks in the workflow did not
work in some cases
+* Fix the problem that the interface of workflow task relationship creation
has no dependency detection
+* Fix the problem that the task cannot be resubmitted when it is fault-tolerant
+* Fix the problem that the prompt information is not clear when the tenant’s
name is too long
+
+# Modify records
+https://github.com/apache/dolphinscheduler/releases/tag/2.0.7
+
+# Download
+https://dolphinscheduler.apache.org/en-us/download/download.html
+
+# Acknowledgement
+This version, like version 2.0.6, is also tempered on the scheduling practice
of 8000+ daily cumulative scheduling jobs in the production environment of
Zhengcai Cloud and fixes the problems feed backed by the community. Special
thanks to the Zhengcai Cloud Data Platform Department for their support of this
release, and to all those who contributed to the release of Apache
DolphinScheduler 2.0.7. It is your unremitting efforts to make the community
progress!
+
+| danielfree | edward-yang | hstdream |
+|------------|--------------|------------|
+| lordk911 | retime123 | zwZjut |
+| Jekong-hao | JinyLeeChina | liqingwang |
+
+How to contribute:
+
+* GitHub Code Repository: https://github.com/apache/dolphinscheduler
+
+* Official Website:https://dolphinscheduler.apache.org/
+
+* Mail List:dev@[email protected]
+
+* Twitter:@DolphinSchedule
diff --git a/blog/en-us/How_can_more_people_benefit_from_big_data.md
b/blog/en-us/How_can_more_people_benefit_from_big_data.md
new file mode 100644
index 0000000000..4dd6a0eca1
--- /dev/null
+++ b/blog/en-us/How_can_more_people_benefit_from_big_data.md
@@ -0,0 +1,210 @@
+---
+title:# How can more people benefit from big data?
+keywords: Apache,DolphinScheduler,scheduler,big
data,ETL,airflow,hadoop,orchestration,dataops,Meetup
+description:During the ApacheCon Asia 2022, Chen Wei, who has more than 10
years of experience in Big Data...
+---
+# How can more people benefit from big data?
+
+<u>During the ApacheCon Asia 2022, Chen Wei, who has more than 10 years of
experience in Big Data development and is head of an offline data factory
development tool from a bank, delivered a keynote speech on how to enable more
people to benefit from Big Data.
+
+Many thanks to volunteer Guan Bo for your text editing work, your contribution
to Apache DolphinScheduler is what keeps the community moving forward!</u>
+
+This presentation focused on the following four key elements.
+
+# Background Information
+Business managers would like access to reliable data to help them make better
business decisions.
+
+End-users want to get their data quickly within a few minutes or hours. They
don’t want to go through the traditional process where they need to submit
requirements to colleagues to the data department and wait weeks to get the
data about requirement changes, development, and online implementation.
+
+Data development teams are getting bigger and more sophisticated to manage
than ever before. Everyone who works with data wants immediate access to
business data to help them make decisions.
+
+Data sources are becoming more complex, such as databases, web pages, logs,
files, ERP, external data, etc.
+
+## User Needs
+Advanced users are looking for limited self-service, with easy, analytical
access to data through self-service.
+
+Application developers only care about the business logic, not the specific
underlying technology, they also don’t want to go deeper into the configuration
associated with platform operations and maintenance.
+
+Hiding as much technical detail as possible, allows business developers as
well as business users to focus more on the business logic and achieving
business goals.
+
+More user-friendly for system administrators, allowing them to quickly and
easily manage the data platform effectively.
+
+Based on the above user requirements, we believe that DataOps is the way to
go, and DataOps built on top of DevOps is the best way to achieve data agility.
+
+## What is DataOps?
+
+DataOps is a collaborative data management practice focused on improving the
communication, integration, and automation of data flows between data
administrators, data consumers, and data developers within an organization.
+
+The goal of DataOps is to deliver value quicker by establishing predictable
delivery as well as managing change to data, data models, and related artifacts.
+
+DataOps uses technology to automate the design, deployment, and management of
data delivery through suitable levels of governance and uses metadata to
improve the availability and value of data in a dynamic environment.
+
+DataOps helps us achieve rapid data innovation, deliver insights to the
business customer at a faster rate, and provide data quality to support data
personnel.
+
+In 2018, DataOps was included in the Technology Maturity Curve for Data
Management, marking the official adoption and rollout of DataOps by the wider
industry. Although DataOps can lower the barriers to data analytics, it does
not make data analytics a simple task.
+
+## Focus of DataOps
+* Rapid innovation and experimentation to deliver better value to customers at
an ever-increasing rate
+* Delivering extremely high-quality data with very low error rates.
+* Delivers interactive value through collaboration between complex arrays of
people as well as technologies and environments.
+* Clarity of measurement and monitoring and transparency of results.
+
+
+## DataOps Core Component
+
+### Job Scheduler
+Provides scheduling of workflows, and offline data development and enables
data developers to focus on the implementation of business logic and improve
development efficiency.
+
+### DevTools
+Mainly provides development tools such as traditional scripting languages SQL,
Python, etc. for quick and easy integration with the scheduling platform.
+
+### Migration and Deployment tools
+Data migration and deployment management tools. Aimed towards industries where
the development and testing environment is network isolated, it requires
effective import and export functions for proper deployment to the production
environment.
+
+### Management and monitoring tools
+Administrator-friendly management, monitoring, and alert tools for offline
tasks related to operations and maintenance.
+
+# Job Scheduler
+Next, we will compare different job schedulers together with you.
+
+## Advantages and disadvantages of mainstream job scheduler
+* **Oozie**
+Oozie is an open-source workflow engine-based framework that provides job
scheduling and coordination for Hadoop MapReduce, and Pig Jobs. Oozie needs to
be deployed to the Java Servlet container to function. It is mainly used for
scheduling tasks at regular intervals and multiple tasks can be scheduled in
sequential logical order of execution.
+
+* **Airflow**
+Airflow is an open-source project for Airbnb’s Workflow, Python-based task
management, job scheduling, and monitoring workflow platform. Airflow is a DAG
(directed acyclic graph) based job scheduling system, which can be interpreted
as an advanced version of crontab, but it solves the task dependency problem
that Crontab cannot handle. Compared to Crontab, Airflow makes it easy to
monitor the status of tasks (if they were executed successfully or not, time of
execution, execution depende [...]
+
+* **Apache Dolphinscheduler**
+DolphinScheduler is a decentralized, scalable and easily visualized DAG
workflow job scheduling platform. Dedicated to solving the complex dependencies
in the data processing process and enabling scheduling systems to be used right
out of the box in the data processing process.
+
+* **Control-M**
+Control-M is a commercial version of a cross-platform job scheduling
management software with powerful features but a lesser degree of
programmability.
+
+* **Azkaban**
+Azkaban is a bulk workflow job scheduler released by Linkedin to execute a set
of jobs and processes within a workflow in a specific order. Azkaban uses job
profiles to establish dependencies between jobs and provides an easy-to-use web
user interface to maintain and track your workflows. Azkaban requires all nodes
to be deployed on a peer-to-peer basis, but under certain circumstances does
not require full node peering as long as it supports high availability.
+
+## Scheduler Services
+* **Timer Service**
+
+Provides a scheduled service of Crontab expressions to execute workflows on a
scheduled cycle.
+
+* **DAG Computation**
+DAG computing often refers to the internal division of a computational job
into several smaller jobs, which are logically related or sequentially
constructed into a DAG (directed acyclic graph) structure.
+
+* **Task Execution**
+Task Execution is run by the scheduling system’s execution engine according to
the type of task, parameters, environment, referenced data source, etc.
+
+* **Environment Manage**
+ Targeting distributed scheduling engines, need to provide node management
capabilities to facilitate task execution at different nodes. This enables
tasks to run at a scale well beyond the limits of a single machine.
+
+* **Alert Service**
+Notify users and system administrators when tasks fail, time out, or are not
completed within a specified time.
+
+* **What’s more?**
+
+In addition to the above, we hope that the scheduler can provide environment
and data source management functions. After the workflow has been configured
with an environment parameter, all tasks can then directly refer to it. Data
sources can be made available for other tasks in an injection method, depending
on the type of task.
+
+## Job Meta
+
+
+* **Task Execution Cycle**
+
+(minutes, hourly, daily, weekly & monthly)
+
+* **Dependency Meta**
+
+Upstream metadata information (files, tables, etc.) depending on the task.
+
+* **Output Meta**
+
+Downstream metadata information (files, tables, etc.) depends on the task.
+
+* **Dependency Meta Classification**
+
+Classify and differentiate the sources of dependent metadata.
+
+* **Dependency Meta Source**
+
+Reverse processing of task-dependent metadata to obtain upstream and
downstream tasks, providing a logical basis for automatic workflow scheduling
and initiation.
+
+* **Job Parallel Execution Info(parallel, serial)**
+
+Controls the parallel execution status of jobs (parallel, serial).
+
+* **Job Type**
+
+Type of job (SQL, SHELL, PYTHON, PROCEDURE, etc.)
+
+## Job Development
+Integration IDE(script language,shell,SQL,python etc)
+
+Integrated IDE development environment (Script language, Shell, SQL, Python
tasks, etc.)
+
+configurable resource file(jar，spark, etc)
+
+Configurable resource files（jar, spark, tasks that cannot be edited using
text, etc.）
+
+custom components
+
+Customizable components, such as off-line data platforms where data
synchronization is a specific component, require only a simple configuration of
data synchronization to implement the synchronization process.
+
+# Third-Party Job
+Third-party job, where a specific application generates a custom script to
accomplish the job goal.
+
+Third-Party Job Integration
+
+There are two methods of integration with third-party jobs, with the second
method being the primary method used.
+
+**First method: pull**
+
+* The job scheduler provides an interface that can be accessed by the
third-party system.
+* Development within the system and configuring the scheduler in the scheduler
system. Configure the job in the third-party system, configure the scheduler in
the scheduling system and execute the job.
+* Single-point of failure and bottleneck for job execution.
+**Disadvantages**: Requires multiple developments in third-party systems and
configuration of the scheduler in the scheduling system.
+
+**Second method: push**
+* The scheduling system provides a programmable API.
+* Third-party systems create jobs and push them to the scheduling system,
where they are automatically configured and scheduled.
+
+## Job Execution And Notification
+
+
+**Job execution**: automatically triggered, based on conditions, upstream task
dependencies
+
+**Job notifications**: progress and status of job execution sent to
third-party systems as a push message
+
+## Sample Configuration For Data Integration
+
+**Sample configurations for third-party system integration are listed above.**
+
+The job scheduling system provides a programmable interface to enable the
injection of environmental information. You only need to specify the different
types of data sources in the third-party system, you do not need to configure
the details of each data source e.g. (IP, port, username, password, etc.), this
part is configured in the scheduling system and the data source and environment
information are injected to the job. The job then parses the data source and
environment from the par [...]
+
+## Job Assignments
+Traditional job scheduling was implemented by users dragging and connecting
links, but we have improved the process by:
+
+* Integrating job scheduling into the workflow.
+* The configuration of upstream and downstream jobs is achieved through
job-dependent metadata information.
+* Some of the tables (some dimension tables) in ETL jobs are immutable and can
be added to a whitelist. The workflow eliminates the need to look for business
logic upstream of the corresponding table in the whitelist when making
automatic changes and resolving dependencies.
+* Views are an important factor affecting workflow automation. Views do not
have ETL jobs, so we need to consider the view as a virtual job and import the
view data into the scheduling platform so that the scheduling system is aware
that the jobs that depend on the view are fundamentally dependent on the tasks
corresponding to the underlying tables in the view.
+
+## Job Impact Analysis
+Thanks to the existence of metadata information, analysis of the impact of
jobs is very straightforward.
+
+* The dependencies of the upstream and downstream metadata are used to resolve
the upstream jobs which are job-dependent, avoiding the use of the manual
configuration to reduce the probability of errors.
+* The calculation of some of the latest impacts, e.g. job suspension, failure,
and impact of downstream jobs.
+
+## Low Code Platform
+* For general situations, it is a good choice to build template jobs that can
be reused to calculate processes for a large number of job configurations,
requiring only a single configuration of the core process to set up the
parameters.
+* Support of specific systems, such as metrics management systems, tagging
systems, data desensitization systems, etc.
+* Coding friendly, able to perform coding when required.
+* User friendly, under certain circumstances, the system coding is used rather
than through user coding, the user can obtain the required data to assemble a
specific data model without having to re-code it, e.g. the user can assemble a
derivative tag based on the base tag by simple click configuration.
+
+# How to contribute:
+
+* GitHub Code Repository: https://github.com/apache/dolphinscheduler
+
+* Official Website:https://dolphinscheduler.apache.org/
+
+* Mail List:dev@[email protected]
+
+* Twitter:@DolphinSchedule
\ No newline at end of file
diff --git
a/blog/en-us/How_did_Yili_explore_a_path_for_digital_transformation_based_on_DolphinScheduler.md
b/blog/en-us/How_did_Yili_explore_a_path_for_digital_transformation_based_on_DolphinScheduler.md
new file mode 100644
index 0000000000..828dbea9d5
--- /dev/null
+++
b/blog/en-us/How_did_Yili_explore_a_path_for_digital_transformation_based_on_DolphinScheduler.md
@@ -0,0 +1,137 @@
+---
+title:# How did Yili explore a “path” for digital transformation based on
DolphinScheduler?
+keywords: Apache,DolphinScheduler,scheduler,big
data,ETL,airflow,hadoop,orchestration,dataops,Meetup
+description:People in general know about Yili. Every year, 1.3 billion...
+---
+# How did Yili explore a “path” for digital transformation based on
DolphinScheduler?
+
+
+# Background and introduction of application
+
+## Introduction of Yili
+People in general know about Yili. Every year, 1.3 billion national consumers
consume an average of 100 million packs of Yili products every day. The
starting point of Yili’s business can be traced back to the cultivation of
grass, the breeding of a cow, the production of a cup of milk, and finally
delivered to consumers through a complex supply chain system, which involves
the primary, secondary and tertiary industries.
+
+## Background of the application
+
+There is a complex application matrix behind the huge business volume, which
poses a great challenge to our technical architecture.
+
+**· The current multi-cloud distribution of applications makes cross-cloud
data migration and multi-cloud unified scheduling a rigid requirement**
+Currently, in the process of cooperation with cloud service providers, Yili
does not only consider the IaaS and PaaS abilities supported by cloud service
providers, but also the ecology behind them, including private domain ecology,
e-commerce, logistics, etc. Therefore, our applications are more distributed on
multiple clouds, and the data generated by these applications are also found on
multiple clouds. The current centralized data structure requires a large amount
of physical data re [...]
+
+· Unified technical architecture to combat entropy increase, reduce cost and
increase efficiency
+
+The first challenge we faced was the ease of use, stability, and scalability
of data integration and scheduling abilities due to massive relocation
requirements. The second one came from the effect of entropy increase brought
by the landing of a large number of applications. In the case of limited
technical resources, it was difficult for us to effectively control our costs
and improve our efficiency, especially in the face of data integration. We were
facing the following core issues:
+
+• Duplicate construction of similar tools and products: Due to a lack of
unified planning, duplicate construction of similar functional products exists.
+
+• Miscellaneous technology selection: AirFlow, Azkaban, Oozie, self-developed
scheduling, etc.
+
+• High construction and derivative costs: costs such as personnel resource
reserves, operation and maintenance, and operation training for multiple
technology stacks.
+
+• Extensibility issues: extensive support for localized individual
requirements.
+
+Before the introduction of Apache DolphinScheduler, our overall technical
architecture was not unified in scheduling and data integration. To meet our
internal rigid requirements, and with the consideration of scalability and
stability, we planned to build a unified scheduling & data integration service
system. After researching a large number of scheduling products, we decided to
use Apache DolphinScheduler as our core engine and make localized
transformations on top of it.
+
+## The positioning and application of the status of the Yili Big Data
Scheduling System
+With limited technical resources, it is challenging to initiate the
development of a tool. We must be clear about who our core users are, what the
core positioning of this product is, and in what circumstances is this product
used.
+
+After sufficient research, we have clarified the core positioning of this
tool. It is not a scheduling platform for simple workflow, but a development
platform of integrated data that suits the multi-cloud business of Yili.
Application developers support the integration of internal and external data,
scheduling requirements of the application, visual orchestration, and
scheduling of data tasks.
+
+Judging from the current application of Yili, the number of daily scheduling
tasks has reached 13,000, there are 15 nodes of clusters and more than 8000
tables are relocated every day.
+
+We have built a unified data platform for data integration, development,
scheduling, operation, and maintenance on the multi-cloud infrastructure for
big data. Through this platform, we can hide the differences between
multi-cloud big data platforms and provide users with a unified development
experience.
+
+**· System Overview of the Data Scheduling Service Platform of Yili
+**
+We have re-abstracted the 2.0.2 version of the entire model of
DolphinScheduler, with the project as the top model, abstracting the three
concepts of resources, roles, and tools, and bound them to the project. Among
them, the toolset is the most important. All the tools used in the data
development process would be expanded here. Users can complete data
integration, development, operation and maintenance, asset management, and
other tasks by authorizing projects in the platform. We have [...]
+
+# Practice & Exploration
+We have upgraded three times based on Apache DolphinScheduler 2.0.2 version
with many function optimization, but there is a logic that runs through our
product design and development process, that is, to meet the core needs of
users to build new functions, improve user experience and to optimize the old
functions. We aim to make the product easy to use, easy to get used to,
user-friendly, and with reasonable moving lines.
+
+## Scenario-oriented data integration
+What is scenario-oriented data integration? Our products are completely open
to users. When developers do data integration, they have a very clear context,
such as synchronizing Mysql data to Hive, ES data to StarRocks, etc. Therefore,
we provide a large number of integration components, which can complete the
integration based on the configuration method. In the whole process, no code is
needed. This improves the efficiency of data integration development.
+
+Before planning the core functions of data integration, we had the following
main problems:
+
+1. Multiple data integration components or technologies make it difficult for
users to choose
+
+2. An inconsistent technical structure that leads to increased maintenance
costs
+
+3. The configuration of data integration is troublesome
+
+4. Poor scalability for secondary development
+
+We implemented the following strategies for these four problems :
+
+1. We only gave users the best choice — there are often no choices to choose
from, so we will just provide the best one.
+
+2. Based on the structure of DataX unified data integration, we solved
problems related to resources and reduced operation and maintenance costs
+
+3. For business applications, we used the template generator to generate task
templates to simplify operations
+
+4. We implemented a reasonable code design
+
+## A lightened asset management
+The core requirement corresponding to this function is as follows: when
applying the native big data capability to build a lightened data warehouse, it
is necessary to gain a clear insight into the technical metadata information,
including library, table, field, and basic attributes. We listed some business
tag information, including subject domains, topics, etc., to facilitate the use
of data by analysts. Based on such demands, we extended this lightened asset
management function on the [...]
+
+**Forward and reverse table building are the core functions**. Forward table
building refers to supporting users to create table models on the platform and
materialize the table models on the big data platform. Reverse table building
is used more often. In many cases, users still use traditional development
methods to write scripts on the ETL server and submit them to the big data
platform for calculation. At this time, the table already exists in advance,
rather than being created visua [...]
+
+Asset retrieval provides asset retrieval functions for development and
business analysis. It can perform lightweight retrieval based on subject
domains, tags, field names, etc., and display asset cards, and upstream and
downstream bloodline information.
+
+## Scheduling of multi-cloud tasks
+Scheduling of multi-cloud tasks is to solve the integrated scheduling problem
of multi-cloud platforms. As mentioned above, Yili’s applications are
distributed in multiple clouds, and this function can complete cross-cloud task
scheduling based on a canvas. We made some extensions and transformations based
on Apache DolphinScheduler. The binding relationship between the environment
and the worker group is flexibly used, and the big data infrastructure is used
to solve the problem of Kerb [...]
+
+We can separately trigger data integration tasks, including data processing
tasks on different clouds. After the processing is completed, data calculation
and business application-oriented planning are triggered on another cloud, and
data services are provided finally. In this way, multi-cloud scheduling is done.
+
+## Optimization of monitoring dashboard
+Based on the original Apache DolphinScheduler dashboard, we did some
optimization and upgrade. By focusing on operation and maintenance, we built
the core indicators that the operation and maintenance work focuses on and
displayed the indicators visually.
+
+The entire monitoring board is divided into task monitoring and cluster
service monitoring.
+
+For task monitoring, we design key indicators like the number of projects,
workflow (online/offline), the number of tasks to be scheduled, the number of
tasks that have been successfully scheduled, an overview of the hourly
operation, the TOP5 statistics on running failures, and the TOP5 statistics on
running time.
+
+For service monitoring, key indicators include CPU, memory, average load,
worker load in each cloud environment, and hourly worker operation overview.
+
+## Upgrade and optimization of users’ experience
+
+We also upgraded and optimized the user experience, such as the optimization
of users’ timed configuration and the overall layout, which aims to optimize
the user experience and make it easier to use.
+
+# Plans
+Lastly, these are the plans for DolphinScheduler.
+
+## Long-term planning
+In the long run, both the community and our company need to have a clear
positioning for DolphinScheduler. It is not a scheduling platform for simple
workflow, but a one-stop intelligent data development platform. There are at
least three important characteristics of this platform:
+
+1. Multi-cloud unification: It can perfectly fit with the new decentralized
data structure, which is a very important direction;
+
+2. Low-code: From the user’s point of view, it can support one-stop visual
modeling, development, and analysis experience for the whole link;
+
+3. Intelligent management: The current data structure is based on reducing the
complexity of data management, metadata-driven governance, and realizing
intelligent data quality management will also be an important proposition.
+
+## Short and mid-range plans:
+Three important points will be focused on in the short term :
+
+1. The combination of cloud-native: The DolphinScheduler community has
provided deployment services based on k8S, so our master, worker, and other
services will be transformed based on containerization;
+
+2. Introduce the testing and online process: add roles such as developers and
reviewers in the project, and control the online actions, and the metadata will
be automatically synchronized after getting online;
+
+3. Quality module: DolphinScheduler 3.0 has added the data quality module. We
will integrate this part and carry out the localized expansion, including
pre-defining data quality, auditing, introducing data blood relationships, etc.
+
+# Reflects on open-source
+During the process of introducing DolphinScheduler, Yili was driven and
inspired by some thoughts on collaboration. That is it’s very important to
build a good collaboration between the company and open-source.
+
+Internally, we will assign specific personnel to follow up on the dynamics of
open-source communities. The R&D team will proactively find bugs and submit
them to the community, and submit PRs after they are fixed.
+
+At the same time, we will also actively interact with the community and check
the trends through weekly and monthly meetings. Companies with limited
technical research and development resources should find an open-source product
that fits the actual use, and truly understand and implement the meaning of
open-source. It is a powerful tool, as it can solve many problems internally,
and notice issues that were never noticed before. Therefore, companies need to
implement a suitable process f [...]
+
+That’s all of my talks. Thank you!
+
+# How to contribute:
+
+* GitHub Code Repository: https://github.com/apache/dolphinscheduler
+
+* Official Website:https://dolphinscheduler.apache.org/
+
+* Mail List:dev@[email protected]
+
+* Twitter:@DolphinSchedule
\ No newline at end of file
diff --git a/site_config/blog.js b/site_config/blog.js
index 37b6a1764d..1da032fe22 100644
--- a/site_config/blog.js
+++ b/site_config/blog.js
@@ -3,6 +3,27 @@ export default {
barText: 'Blog',
postsTitle: 'All posts',
list: [
+ {
+ title: 'How did Yili explore a “path” for digital
transformation based on DolphinScheduler?,
+ author: 'Debra Chen',
+ dateStr: '2022-10-31',
+ desc: 'People in general know about Yili. Every year, 1.3
billion... ',
+ link:
'/en-us/blog/How_did_Yili_explore_a_path_for_digital_transformation_based_on_DolphinScheduler.html',
+ },
+ {
+ title: 'How can more people benefit from big data?',
+ author: 'Debra Chen',
+ dateStr: '2022-10-29',
+ desc: 'During the ApacheCon Asia 2022, Chen Wei, who has more
than 10 years... ',
+ link:
'/en-us/blog/How_can_more_people_benefit_from_big_data.html',
+ },
+ {
+ title: 'Apache DolphinScheduler 2.0.7 released, fixes
complement and fault tolerance',
+ author: 'Debra Chen',
+ dateStr: '2022-10-28',
+ desc: 'Recently, Apache DolphinScheduler version 2.0.7 was
officially released...... ',
+ link: '/en-us/blog/Apache_DolphinScheduler_2.0.7.html',
+ },
{
title: 'Apache DolphinScheduler 3.0.1 released, with the
scheduling kernel and UI optimized',
author: 'Debra Chen',
diff --git a/site_config/home.jsx b/site_config/home.jsx
index 9517747d87..5a7c619b45 100644
--- a/site_config/home.jsx
+++ b/site_config/home.jsx
@@ -558,6 +558,13 @@ export default {
events: {
title: 'Events & News',
list: [
+ {
+ img: 'https://miro.medium.com/max/630/0*Ep9rZFSTviU8SjmP',
+ title: 'Apache DolphinScheduler 2.0.7 released, fixes complement and
fault tolerance',
+ content: 'Recently, Apache DolphinScheduler version 2.0.7 was
officially released...',
+ dateStr: '2022-9-28',
+ link: '/en-us/blog/Apache_DolphinScheduler_2.0.7.html',
+ },
{
img: 'https://miro.medium.com/max/1400/1*zgkZk3lhn3UhLAKkHM9wzQ.png',
title: 'ApacheCon Asia 2022 Review | Application of DolphinScheduler
in T3Go One-stop Platform',
@@ -574,14 +581,6 @@ export default {
link: '/en-us/blog/Apache_dolphinScheduler_3.0.0.html',
},
- {
- img: 'https://miro.medium.com/max/630/0*Ep9rZFSTviU8SjmP',
- title: 'Apache DolphinScheduler Extends Capabilities Through Python
API and AWS Support',
- content: 'In the ever-changing world of technology, data is
abundant. In fact,....',
- dateStr: '2022-8-26',
- link: '/en-us/blog/Python_API_and_AWS_Support.html',
- },
-
],
},
userreview: {
