Re: Apache Spark and Apache Ignite

2019-11-05 Thread Wellington Alves das Neves
Hi,

1. What's your main goal?
   We need to run Genetic Algorithms in parallel to return the best
execution plan for a factory production line.

To solve optimization task on the data in Ignite or in Spark?
   As I have little experience, I'm evaluating the best architecture we
can be applying to this scenario,
   and remembering that we will perform using AWS Elastic MapReduce
(EMR) service.

Regarding architecture, could you suggest this scenario?


2. What's the average size of initial population?
 We are evaluating the size.


Thks!



Em ter., 5 de nov. de 2019 às 08:24, zaleslaw 
escreveu:

> Hi, Wellington
>
> I'll be happy to help you if you give me more information of your goal.
>
> I have a few questions, could you answer please?
>
> 1. What's your main goal? To solve optimization task on the data in Ignite
> or in Spark?
> 2. What's the average size of initial population?
> 3. Did you run these  examples
> <
> https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml/genetic>
>
> to solve, for example knapsack problem?
> 4. Did you have a look here, docs
> https://apacheignite.readme.io/docs/genetic-algorithms
>
> Yes, Apache Ignite and Apache Spark has integration  bridge
> <https://apacheignite-fs.readme.io/docs>   which give us ability to use
> Ignite instead of .cache() or persist() to keep dataframes in-memory for
> intermediate calculations.
>
> But we have no support for GA framework or another ML parts as part of
> extended Spark API.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>


-- 
Wellington Alves das Neves
*Software Engineer*

+55 (17) 99194-7119
wellingtonalvesne...@gmail.com
[image: linkedin] <http://www.linkedin.com/in/wellingtonalvesneves> [image:
skype] <http://wellingtonalvesne...@gmail.com/> [image: gitlab]
<https://gitlab.com/WANeves>


Apache Spark and Apache Ignite

2019-11-05 Thread Wellington Alves das Neves
Hi,

I am currently researching an architecture on AWS - Elastic MapReduce (EMR)
to run Genetic Algorithms (GA).
Apache Ignite already has some genetic algorithms (GA) defined, so we would
like to do some testing with it integrated with Apache Spark.

But I found little material on how to implement this Apache Ignite GA
architecture along with Apache Spark.

Thks!

-- 
Wellington Alves das Neves
*Software Engineer*

+55 (17) 99194-7119
wellingtonalvesne...@gmail.com
[image: linkedin] <http://www.linkedin.com/in/wellingtonalvesneves> [image:
skype] <http://wellingtonalvesne...@gmail.com/> [image: gitlab]
<https://gitlab.com/WANeves>