Hi Tony: Thanks for the detailed response, it's very helpful.
Being very focused on the Hadoop stack for the last 5- years, I had not heard of the term SMACK before. I'm familiar with all but akka so I'm reading up on that. I see virtual sensors in the source tree. Are there specs for IoT devices that should be implemented as a virtual sensor to help show the possibilities with iota? In this context, "spec" refers to some public data source that can generate data that can mimic data via a virtual sensor; for example, weather data or something like that. Thanks, Gunnar On Sat, Feb 11, 2017 at 1:06 AM, Tony Faustini <[email protected]> wrote: > I meant SMACK Stack IoT Platform in the original posting …. > > On Feb 10, 2017, at 11:49 PM, Tony Faustini <[email protected]> wrote: > > This is a great question. Our intension is for iota to be a SMACK Stack > Not Platform. This means it uses > > 1) Apache Spark > 2) Apache Mesos > 3) Akka > 4) Apache Cassandra > 5) Apache Kafka > > The Fey engine is a component of the iota platform but not the entire > platform. The platform is a vertical stack that has a Mesos Cluster as it’s > highest layer. This could be in the cloud (AWS, Azure, …). The next level > of the platform might be at the device level say a RaspberryPi with sensors > or an aggregation server. The edge devices can process data locally on the > device using Fey components or/and feed them up the stack using Fey > components. Large amounts of data will arrive at the Mesos cluster. This > data can be proceeded by Fey and Fey components running as an Akka cluster > on Mesos. It can also be processed using a Spark cluster also running on > the Mesos layer. Cassandra is used to persist data at the cluster level. > Kafka is the message bus for major software component to use when > communicating with each other. The SMACK stack becomes the data pipeline > and processing engine for IoT data streaming from edge based sensors. > > The first component of iota that we are making available is the Fey engine > and a selection of Fey components. For example a Fey component for Kafka > that includes Kafka producer and consumer capabilities. The iota Fey engine > can run at all levels of the iota stack. This can include a RaspberryPi on > a JVM in standalone Akka mode or on the Mesos Cluster where it runs as an > Akka cluster. > > I hope this sheds light on the big picture. We can discuss the role of > Docker and Spark later so let’s not worry about them right now. > > This is a grand vision so where do we start? The simplest way to start is > with the Fey engine and Fey performers. > > We are first releasing a version of the Fey engine and a selection of > performers. Interested parties can download and experiment with these. We > hope they will be inspired to build their own software components that they > can use and share with others. The best components can be included in the > Fey component release. A rich set of these components (called performers) > will enable powerful applications to be assembled by others providing data > pipeline and processing functionality needed in the iota platform. Here are > a few examples of performers that could be and have been written > 1) A Restful performer that implements get, put and post to reach out to > get data and control “Things” on the internet. This could run on a > RaspberryPi and data could be passed on to other performers. > 2) A component that ingests data from other performers and processes the > data based on some mathematical or statistical decisions and selectively > outputs that data to other performers. > 3) A component that interacts get/put to an Apache Ignite in-memory > system. Data received from other performers can be put the data store or > data read from the store can be sent to other performers. > 4) A component that interacts with Apache ActiveMQ similar to 3) > 5) A component that receives a high level image from another component, > makes it low resolution image version and send it on to other performers. > 6) A component that can read values from I2C sensors and pass the value > read to other components. > > These are some examples of component that have been and can be written as > Fey Performers. They form the core of a communication and processing system > that can be used to define a component based application. Such applications > play an important role in the iota platform. > > How can someone contribute to the code base? > > 1) Select you preferred development workflow. The following are all free > - vim and sbt (simple build tool) on the command line > - IntelliJ Community edition (sbt) > - Eclipse (sbt) > > my personal preference is IntelliJ Community edition - Eclipse is also a > valid option if you prefer Eclipse. > > Follow the instructions that are being placed in the iota wiki on how to > download and run the fey engine and fey performers. Think of development > for Fey in the following terms > 1) You have a palette of performers with different functionalities > 2) Your applications will be limited by the set of performers you have > available to you. Say you have three performers P1 that reads numbers form > a sensor at a certain frequency say 5 seconds, you have P2 that waits for > values coming from other performers and if those values are between 0 and > 100 they are set to other performers, and you have P3 that receives data > from other performers and places a message in the log file. > 3) To create an application you would create the following orchestration > P1 -> P2 -> P3 (it is a json object that defines each of the components > used how they interact with one another P1->P2->P3 and any parameters need > by each performer e.g. the 5 seconds frequency for this instance of > performer P1). > 4) You may find yourself needing to write an application for which you do > not have a complete set of components. Say it’s the example in 3) above but > you also want to send the message on a Kafka topic? You would then have to > either find a pre-existing Kafka performer that you get from some else or > you would write your own. > > So get familiar with Fey and Fey components on your own local dev machine > and write a simple Fey Application (orchestration) if you don’t have the > needed functionality write one or more performs to give you the > functionality you need. > This would be a good start. For run put Fey on a RaspberryPI or some other > Linux based ARM device (Intel Edition?) and run Fey on the device. > > I hope my long winded explanation is of some help. Feel free to ask other > questions if I have not explained things well or not answered them at all. > > -Tony > > > On Feb 10, 2017, at 10:13 PM, Gunnar Tapper <[email protected]> > wrote: > > Hi, > > I'm trying to match the code with what's being said on > http://iota.incubator.apache.org/features.html > > I'm not clear where I find; for example, docker and spark parts? > > What's your advise for someone who wants to contribute? It seems like a > good idea to read up on akka but then what? > > I'm setting up my project to use eclipse. Is that the IDE of choice or is > there something more appropriate? > > > -- > Thanks, > > Gunnar > *If you think you can you can, if you think you can't you're right.* > > > > -- Thanks, Gunnar *If you think you can you can, if you think you can't you're right.*
