This is an automated email from the ASF dual-hosted git repository. glauesppen pushed a commit to branch feat/blog-federated-ai in repository https://gitbox.apache.org/repos/asf/incubator-wayang-website.git
commit 2f5650e6b7f77d810e86215ebabc223c7e090f68 Author: Glaucia Esppenchutz <[email protected]> AuthorDate: Wed Apr 17 22:31:03 2024 +0100 Add federated ai blog post --- blog/2024-04-17-federated-ai.md | 73 +++++++++++++++++++++ blog/authors.yml | 5 ++ .../architecture/federated-ai-architecture-1.png | Bin 0 -> 193652 bytes .../architecture/federated-ai-architecture-2.png | Bin 0 -> 217109 bytes 4 files changed, 78 insertions(+) diff --git a/blog/2024-04-17-federated-ai.md b/blog/2024-04-17-federated-ai.md new file mode 100644 index 00000000..767d9fab --- /dev/null +++ b/blog/2024-04-17-federated-ai.md @@ -0,0 +1,73 @@ +--- +slug: wayang-federated-ai +title: Wayang and the Federated AI +authors: [glauesppen] +tags: [wayang, federated, ai] +--- + +# The Federated AI + +AI systems and applications are widely used nowadays, from assisting grammar spellings to +detecting early signs of cancer cells. Building an AI requires a lot of data and training to achieve +the desired results, and federated learning is an approach to make AI training more viable. +Federated learning (or collaborative learning) is a technique that trains AI models on data +distributed across multiple serves or devices. It does so without centralizing data on a single +place or storage. It also prevents the possibility of data breaches and protects sensitive +personal data. One of the significant challenges in working with AI is the variety of tools found +in the market or the open-source community. Each tool provides results in a different form; +integrating them can be pretty challenging. Let's talk about Apache Wayang (incubating) and +how it can help to solve this problem. + +## Apache Wayang in the Federated AI world + +Apache Wayang (Wayang, for short), a project in an incubation phase at Apache Software +Foundation (ASF), integrates big data platforms and tools by removing the complexity of +worrying about low-level details. Interestingly, even if it was not designed for, Wayang could +also serve as a scalable platform for federated learning: the Wayang community is starting to +work on integrating federated learning capabilities. In a federated learning approach, Wayang +would allow different local models to be built and exchange its model results across other data +centers to combine them into a single enhanced model. + + +## A real-world example + +Let's consider a real-world scenario. Hospitals and health organizations have increased their +investments in machine/deep learning initiatives to learn more and predict diagnostics. +However, due to legal frameworks, sharing patients' information or diagnostics is impossible, +and the solution would be to apply federated learning. To solve this problem, we could use +Wayang to help to train the models. See the diagram 1 below: + +<br/> +<img width="75%" alt="wayang stack" src="/img/architecture/federated-ai-architecture-1.png" /> +<br/><br/> + +As a first step, the data scientists would send an ML task to Wayang, which will work as an +abstraction layer to connect to different data processing platforms, sparing the time to build +integration code for each. Then, the data platforms process and generate the results that will +be sent back to Wayang. Wayang aggregates the results into one "global result" and sends it +back to the requestor as a next step. + +<br/> +<img width="75%" alt="wayang stack" src="/img/architecture/federated-ai-architecture-2.png" /> +<br/><br/> + +The process repeats until the desired results are achieved. +Although it is very much like a Federated learning pipeline, Wayang removes a considerable +layer of complexity from the developers by integrating with diverse types of data platforms. It +also brings fast development and reduces the need for a deep understanding of data +infrastructure or integrations. Developers can focus on the logic and how to execute tasks +instead of details about data processors. + +### Follow Wayang + +Apache Wayang is in an incubation phase and has a potential roadmap of implementations +coming soon (including the federated learning aspect as well as an SQL interface and a novel +data debugging functionality). If you want to hear or join the community, consult the link +https://wayang.apache.org/community/ , join the mailing lists, contribute with new ideas, +write documentation, or fix bugs. + +<br/> + +##### Thank you! +I (Gláucia) want to thank professor Jorge Quiané for the guidance to write this blog post. +Thanks for incentivate me to join the project and for the knowledge shared. I will always remember you. \ No newline at end of file diff --git a/blog/authors.yml b/blog/authors.yml index 6a265169..2bde36eb 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -18,4 +18,9 @@ juripetersen: title: Apache Committer url: https://github.com/juripetersen image_url: https://avatars.githubusercontent.com/u/43411515?v=4 +glauesppen: + name: Gláucia Esppenchutz + title: (P)PMC Apache Wayang + url: https://github.com/glauesppen + image_url: https://avatars.githubusercontent.com/glauesppen diff --git a/static/img/architecture/federated-ai-architecture-1.png b/static/img/architecture/federated-ai-architecture-1.png new file mode 100644 index 00000000..31c137c7 Binary files /dev/null and b/static/img/architecture/federated-ai-architecture-1.png differ diff --git a/static/img/architecture/federated-ai-architecture-2.png b/static/img/architecture/federated-ai-architecture-2.png new file mode 100644 index 00000000..c316f0de Binary files /dev/null and b/static/img/architecture/federated-ai-architecture-2.png differ
