[
https://issues.apache.org/jira/browse/IGNITE-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Dmitriev updated IGNITE-7437:
-----------------------------------
Attachment: Снимок экрана 2018-01-21 в 20.57.27.png
> Partition based dataset implementation
> --------------------------------------
>
> Key: IGNITE-7437
> URL: https://issues.apache.org/jira/browse/IGNITE-7437
> Project: Ignite
> Issue Type: New Feature
> Components: ml
> Reporter: Yury Babak
> Assignee: Anton Dmitriev
> Priority: Major
> Attachments: Снимок экрана 2018-01-21 в 20.56.57.png, Снимок экрана
> 2018-01-21 в 20.57.16.png, Снимок экрана 2018-01-21 в 20.57.27.png
>
>
> We want to implement our dataset based on entire partition instead of key
> sets.
>
> *A main idea behind the partition based datasets is the classic
> [MapReduce.|https://en.wikipedia.org/wiki/MapReduce]*
> The most important advantage of the MapReduce is an ability to perform
> computations on a data distributed across the cluster without involving
> significant data transmissions over the network. This idea is adopted in the
> partition based datasets in the following way:
> 1. Every dataset or learning context consists of partitions.
> 2. Partitions are built on top of the Apache Ignite Cache partitions (as a
> primary storage).
> 3. Computations needed to be performed on a dataset or learning context
> splits on Map operations which executes on every partition and Reduce
> operations which reduces results of Map operations into one final result.
> _Why partitions have been selected as a building block of dataset and
> learning contain instead of cluster node?_
> One of the fundamental ideas of Apache Ignite Cache is that partitions are
> atomic, which means that they cannot be splitted between multiply nodes. As
> result in case of rebalancing or node failure partition will be recovered on
> another node with the same data it contained on the previous node.
> In case of machine learning algorithm it's very important because most of the
> ML algorithms are iterative and require some context maintained between
> iterations. This context cannot be split or merged and should be maintained
> in the consistent state during the whole learning process.
> *Another idea behind the partition based datasets is that we need to have
> data (in every partition) in
> [BLAS-|https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms]like
> format as much as it possible.*
> [BLAS|https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms] and
> [CUDA|https://en.wikipedia.org/wiki/CUDA] makes machine learning 100x faster
> and more reliable than algorithms based on self-written linear algebra
> subroutines and it means that not using BLAS is a recipe for disaster. In
> other words we need to keep data in BLAS-like format at any price.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
