I was really hoping it came through ok, Oh well :) Here’s an image form: http://imgur.com/a/DUDsD
> On Apr 13, 2017, at 4:05 PM, Segerlind, Nathan L > <[email protected]> wrote: > > The diagram became garbled in the text format. > Could you resend it as a pdf? > > Thanks, > Nate > > -----Original Message----- > From: Nathanael Smith [mailto:[email protected]] > Sent: Thursday, April 13, 2017 4:01 PM > To: [email protected]; [email protected]; > [email protected] > Subject: [Discuss] - Future plans for Spot-ingest > > How would you like to see Spot-ingest change? > > A. continue development on the Python Master/Worker with focus on performance > / error handling / logging B. Develop Scala based ingest to be inline with > code base from ingest, ml, to OA (UI to continue being ipython/JS) C. Python > ingest Worker with Scala based Spark code for normalization and input into DB > > Including the high level diagram: > +------------------------------------------------------------------------------------------+ > | +--------------------------+ > +-----------------+ | > | | Master | A. B. C. | Worker > | | > | | A. Python +---------------+ A. | A. Python > | | > | | B. Scala | | +-------------> > +----+ | > | | C. Python | | | | > | | | > | +---^------+---------------+ | | > +-----------------+ | | > | | | | | > | | > | | | | | > | | > | | +Note--------------+ | | > +-----------------+ | | > | | |Running on a | | | | Spark > Streaming | | | > | | |worker node in | | | B. C. | B. Scala > | | | > | | |the Hadoop cluster| | | +--------> C. Scala > +-+ | | > | | +------------------+ | | | | > | | | | > | A.| | | | > +-----------------+ | | | > | B.| | | | > | | | > | C.| | | | > | | | > | +----------------------+ +-v------+----+----+-+ > +--------------v--v-+ | > | | | | | | > | | > | | Local FS: | | hdfs | | Hive > / Impala | | > | | - Binary/Text | | | | - > Parquet - | | > | | Log files - | | | | > | | > | | | | | | > | | > | +----------------------+ +--------------------+ > +-------------------+ | > +------------------------------------------------------------------------------------------+ > > Please let me know your thoughts, > > - Nathanael > > >
