Another thing you can use is the java port of nupic and will be able use it with not so much on Spark / Haddop. Spark will be good choice as able to exploit memory better or you can look into Apache Flink if you want to play with more fancier thing ;)
- Gurvinder On 06/04/2015 04:46 PM, Michael Parco wrote: > I think integrating nupic into existing big data platforms would be > extremely beneficial. Currently I'm working on implementing nupic into > pyspark and running on data that is being passed by kafka and spark > streaming. > > As for running in batch in big data, same type idea. There is already > pydoop which integrates regular python scripts to read data from hdfs, > so integrating this with nupic could at least give you access to a > distributed file store, past that it is finding a way to distribute the > load in a cluster. > > On Thu, Jun 4, 2015 at 9:16 AM, Katerina Muradyan > <[email protected] <mailto:[email protected]>> wrote: > > I agree with Pulin that this is very interesting subject. Even from > the philosophical point of view. > > Correct me if I am wrong (really) that NuPic and Big Data concept > are not so far one from another. Because the principle of Big Data > is to discover patterns in big amount of information which results > into significant analysis. > > I am not a developer so I am not strong with the engineering part, > and maybe what I say already exists.. But I think that the main > interest of this union is not just to build another Big Data machine > based on NuPic platform. But to enable this program to go even > further and analyse what it needs as information, and actually how > to adopt new information by itself. Actually to teach the program to > be curious and to search for things that are not even programmed in it. > > Would love to try to work on it! > > Katja > > > > > 2015-06-04 5:33 GMT+02:00 Austin Marshall <[email protected] > <mailto:[email protected]>>: > > I would /love/ to see some nupic spark integration, personally. > >> On Jun 3, 2015, at 9:54 AM, Matthew Taylor <[email protected] >> <mailto:[email protected]>> wrote: >> >> Aha, so you're just talking about "big data" in general. Yes, >> using >> NuPIC in an architecture like this is a huge potential for its >> usage. >> I do not think there are any sample applications of this, and >> I'm not >> sure if anyone is currently doing it. >> >> Is anyone reading this toying with NuPIC in architectures like >> Hadoop, >> Storm, or Spark? >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> >> On Wed, Jun 3, 2015 at 8:39 AM, Pulin Agrawal >> <[email protected] <mailto:[email protected]>> wrote: >>> Román, >>> >>> I am very interested in both of these things. I was planning >>> to learn Big >>> Data over the summer. So, I was also thinking if I could use >>> NuPIC in that >>> architecture. I still need to learn a lot about both of these >>> things. I have >>> a good understanding of the whitepaper on CLA but I haven't >>> used the NuPIC >>> platform to build anything yet. But maybe working on this >>> kind of a thing >>> will help me in learning both these things together. >>> >>> Matt, >>> >>> Big Data Architecture from my naive understanding is about >>> dealing with >>> large amounts of fast streaming data ('Big Data') using a >>> distributed file >>> system (I think they call it Hadoop) using a MapReduce Algorithm. >>> >>> I don't understand MapReduce very well but I think overall it >>> is a two step >>> process Map algorithm maps the data to various processing >>> nodes and Reduce >>> algorithm obtains results from these nodes and compiles the >>> result. >>> >>> Pulin Agrawal >>> पुलिन अग्रवाल >>> >>> On Tue, Jun 2, 2015 at 5:41 PM, Matthew Taylor >>> <[email protected] <mailto:[email protected]>> wrote: >>>> >>>> Román, >>>> >>>> Can you tell us more about this "Big Data Architecture"? Is >>>> it some >>>> other open source project? >>>> --------- >>>> Matt Taylor >>>> OS Community Flag-Bearer >>>> Numenta >>>> >>>> >>>> On Sun, May 31, 2015 at 1:40 PM, Román Martín González >>>> <[email protected] <mailto:[email protected]>> wrote: >>>>> Hello, >>>>> >>>>> I am very interested in integrating the Nupic Algorithms >>>>> and the Big >>>>> Data >>>>> Architecture, and create a Streaming and Batch >>>>> BigData/NuPIC Solution as >>>>> Open Source project. >>>>> >>>>> I would like to know if there is already a similar project >>>>> or if the >>>>> NuPIC >>>>> team is interested in this project? >>>>> >>>>> Best regards, thank you very much. >>>>> >>>>> Roman Martin >>>> >>> >> > > > > > -- > Katerina Muradyan > 06 50 20 13 71 > [email protected] <mailto:[email protected]> > > 2015-06-04 5:33 GMT+02:00 Austin Marshall <[email protected] > <mailto:[email protected]>>: > > I would /love/ to see some nupic spark integration, personally. > >> On Jun 3, 2015, at 9:54 AM, Matthew Taylor <[email protected] >> <mailto:[email protected]>> wrote: >> >> Aha, so you're just talking about "big data" in general. Yes, >> using >> NuPIC in an architecture like this is a huge potential for its >> usage. >> I do not think there are any sample applications of this, and >> I'm not >> sure if anyone is currently doing it. >> >> Is anyone reading this toying with NuPIC in architectures like >> Hadoop, >> Storm, or Spark? >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> >> On Wed, Jun 3, 2015 at 8:39 AM, Pulin Agrawal >> <[email protected] <mailto:[email protected]>> wrote: >>> Román, >>> >>> I am very interested in both of these things. I was planning >>> to learn Big >>> Data over the summer. So, I was also thinking if I could use >>> NuPIC in that >>> architecture. I still need to learn a lot about both of these >>> things. I have >>> a good understanding of the whitepaper on CLA but I haven't >>> used the NuPIC >>> platform to build anything yet. But maybe working on this >>> kind of a thing >>> will help me in learning both these things together. >>> >>> Matt, >>> >>> Big Data Architecture from my naive understanding is about >>> dealing with >>> large amounts of fast streaming data ('Big Data') using a >>> distributed file >>> system (I think they call it Hadoop) using a MapReduce Algorithm. >>> >>> I don't understand MapReduce very well but I think overall it >>> is a two step >>> process Map algorithm maps the data to various processing >>> nodes and Reduce >>> algorithm obtains results from these nodes and compiles the >>> result. >>> >>> Pulin Agrawal >>> पुलिन अग्रवाल >>> >>> On Tue, Jun 2, 2015 at 5:41 PM, Matthew Taylor >>> <[email protected] <mailto:[email protected]>> wrote: >>>> >>>> Román, >>>> >>>> Can you tell us more about this "Big Data Architecture"? Is >>>> it some >>>> other open source project? >>>> --------- >>>> Matt Taylor >>>> OS Community Flag-Bearer >>>> Numenta >>>> >>>> >>>> On Sun, May 31, 2015 at 1:40 PM, Román Martín González >>>> <[email protected] <mailto:[email protected]>> wrote: >>>>> Hello, >>>>> >>>>> I am very interested in integrating the Nupic Algorithms >>>>> and the Big >>>>> Data >>>>> Architecture, and create a Streaming and Batch >>>>> BigData/NuPIC Solution as >>>>> Open Source project. >>>>> >>>>> I would like to know if there is already a similar project >>>>> or if the >>>>> NuPIC >>>>> team is interested in this project? >>>>> >>>>> Best regards, thank you very much. >>>>> >>>>> Roman Martin >>>> >>> >> > > > > > -- > Katerina Muradyan > 06 50 20 13 71 > [email protected] <mailto:[email protected]> > >
