I think integrating nupic into existing big data platforms would be extremely beneficial. Currently I'm working on implementing nupic into pyspark and running on data that is being passed by kafka and spark streaming.
As for running in batch in big data, same type idea. There is already pydoop which integrates regular python scripts to read data from hdfs, so integrating this with nupic could at least give you access to a distributed file store, past that it is finding a way to distribute the load in a cluster. On Thu, Jun 4, 2015 at 9:16 AM, Katerina Muradyan <[email protected]> wrote: > I agree with Pulin that this is very interesting subject. Even from the > philosophical point of view. > > Correct me if I am wrong (really) that NuPic and Big Data concept are not > so far one from another. Because the principle of Big Data is to discover > patterns in big amount of information which results into significant > analysis. > > I am not a developer so I am not strong with the engineering part, and > maybe what I say already exists.. But I think that the main interest of > this union is not just to build another Big Data machine based on NuPic > platform. But to enable this program to go even further and analyse what it > needs as information, and actually how to adopt new information by itself. > Actually to teach the program to be curious and to search for things that > are not even programmed in it. > > Would love to try to work on it! > > Katja > > > > > 2015-06-04 5:33 GMT+02:00 Austin Marshall <[email protected]>: > >> I would *love* to see some nupic spark integration, personally. >> >> On Jun 3, 2015, at 9:54 AM, Matthew Taylor <[email protected]> wrote: >> >> Aha, so you're just talking about "big data" in general. Yes, using >> NuPIC in an architecture like this is a huge potential for its usage. >> I do not think there are any sample applications of this, and I'm not >> sure if anyone is currently doing it. >> >> Is anyone reading this toying with NuPIC in architectures like Hadoop, >> Storm, or Spark? >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> >> On Wed, Jun 3, 2015 at 8:39 AM, Pulin Agrawal <[email protected]> >> wrote: >> >> Román, >> >> I am very interested in both of these things. I was planning to learn Big >> Data over the summer. So, I was also thinking if I could use NuPIC in that >> architecture. I still need to learn a lot about both of these things. I >> have >> a good understanding of the whitepaper on CLA but I haven't used the NuPIC >> platform to build anything yet. But maybe working on this kind of a thing >> will help me in learning both these things together. >> >> Matt, >> >> Big Data Architecture from my naive understanding is about dealing with >> large amounts of fast streaming data ('Big Data') using a distributed file >> system (I think they call it Hadoop) using a MapReduce Algorithm. >> >> I don't understand MapReduce very well but I think overall it is a two >> step >> process Map algorithm maps the data to various processing nodes and Reduce >> algorithm obtains results from these nodes and compiles the result. >> >> Pulin Agrawal >> पुलिन अग्रवाल >> >> On Tue, Jun 2, 2015 at 5:41 PM, Matthew Taylor <[email protected]> wrote: >> >> >> Román, >> >> Can you tell us more about this "Big Data Architecture"? Is it some >> other open source project? >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> >> On Sun, May 31, 2015 at 1:40 PM, Román Martín González >> <[email protected]> wrote: >> >> Hello, >> >> I am very interested in integrating the Nupic Algorithms and the Big >> Data >> Architecture, and create a Streaming and Batch BigData/NuPIC Solution as >> Open Source project. >> >> I would like to know if there is already a similar project or if the >> NuPIC >> team is interested in this project? >> >> Best regards, thank you very much. >> >> Roman Martin >> >> >> >> >> >> > > > -- > Katerina Muradyan > 06 50 20 13 71 > [email protected] > > 2015-06-04 5:33 GMT+02:00 Austin Marshall <[email protected]>: > >> I would *love* to see some nupic spark integration, personally. >> >> On Jun 3, 2015, at 9:54 AM, Matthew Taylor <[email protected]> wrote: >> >> Aha, so you're just talking about "big data" in general. Yes, using >> NuPIC in an architecture like this is a huge potential for its usage. >> I do not think there are any sample applications of this, and I'm not >> sure if anyone is currently doing it. >> >> Is anyone reading this toying with NuPIC in architectures like Hadoop, >> Storm, or Spark? >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> >> On Wed, Jun 3, 2015 at 8:39 AM, Pulin Agrawal <[email protected]> >> wrote: >> >> Román, >> >> I am very interested in both of these things. I was planning to learn Big >> Data over the summer. So, I was also thinking if I could use NuPIC in that >> architecture. I still need to learn a lot about both of these things. I >> have >> a good understanding of the whitepaper on CLA but I haven't used the NuPIC >> platform to build anything yet. But maybe working on this kind of a thing >> will help me in learning both these things together. >> >> Matt, >> >> Big Data Architecture from my naive understanding is about dealing with >> large amounts of fast streaming data ('Big Data') using a distributed file >> system (I think they call it Hadoop) using a MapReduce Algorithm. >> >> I don't understand MapReduce very well but I think overall it is a two >> step >> process Map algorithm maps the data to various processing nodes and Reduce >> algorithm obtains results from these nodes and compiles the result. >> >> Pulin Agrawal >> पुलिन अग्रवाल >> >> On Tue, Jun 2, 2015 at 5:41 PM, Matthew Taylor <[email protected]> wrote: >> >> >> Román, >> >> Can you tell us more about this "Big Data Architecture"? Is it some >> other open source project? >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> >> On Sun, May 31, 2015 at 1:40 PM, Román Martín González >> <[email protected]> wrote: >> >> Hello, >> >> I am very interested in integrating the Nupic Algorithms and the Big >> Data >> Architecture, and create a Streaming and Batch BigData/NuPIC Solution as >> Open Source project. >> >> I would like to know if there is already a similar project or if the >> NuPIC >> team is interested in this project? >> >> Best regards, thank you very much. >> >> Roman Martin >> >> >> >> >> >> > > > -- > Katerina Muradyan > 06 50 20 13 71 > [email protected] >
