Hi Nicolaas, I have always been curious about the data model of OAE, which uses NoSQL database (Cassandra in this case). Can you give me some pointers on that?
I believe many people are quite new to NoSQL databases and different ways of data modeling for NoSQL vs. Relational database may be a barrier for community contributions for backend development. Thanks, Harry On Oct 18, 2012, at 7:25 PM, Nicolaas Matthijs wrote: > Hi everyone, > > As most of you know, a number of Sakai OAE stakeholder organizations have > withdrawn from the managed project over the last 6 months. Key reasons cited > for these departures include: > > - Concerns about the scalability and viability of the Nakamura architecture > - Slower than desired progress on feature development > - Misalignments between stakeholders > > These withdrawals have had a significant negative impact on the level of > financial and in-kind contributions underwriting the development team. In > addition, reports from piloting institutions together with performance test > data produced by the team have made it clear that OAE’s performance and > scalability characteristics leave much to be desired. While sufficient for > small-scale pilots involving a few thousand users, OAE’s performance under > load is wholly insufficient for adopters considering an enterprise production > deployment supporting tens of thousands of users and above, including the > web-scale deployments we are likely to see in the future. Upgrading between > versions also remains a challenging exercise for operations teams. While the > remaining stakeholders believe that the concepts embodied in OAE’s design > work remain relevant and valuable, current circumstances dictate that we > adopt a new technical approach. > > In late August we recommended to the project stakeholders that the managed > project cease development on OAE Nakamura, setting aside its technical debt > in favor of a re-engineered back-end that re-positions the project as a > multi-tenant, highly scalable, cloud-ready application. The strategy that we > set out involves an initial and temporary reduction in application scope; > existing features will be disabled until their corresponding back-end > services have been recreated. Our proposal was accepted, and in early > September a reconstructed and considerably leaner project team commenced work > on providing a new back-end architecture written in node.js and using Apache > Cassandra for persistence.[1] The code is licensed ECL 2.0 and is publicly > available in a Github repo named “Hilary”.[2] Additional repos have been > created for the test data model loader, Tsung performance tests, nightly > performance test scripts and puppet deployment scripts.[3] > > We have retained OmniTI, a web performance and scalability consultancy, to > help guide our work. A series of one-week sprints have been organized in > order to keep the pace of development high as well as allowing for quick > resets under the “fail fast” rule. We start from a set of purposefully > designed and documented REST APIs (see attachment). Each REST API and its > accompanying internal service will include unit and integration tests. The > team will set up and maintain a reference deployment in the cloud, on which > extensive nightly performance tests will be run. The existing UI, benefiting > from a number of client-side performance improvements included in the OAE > 1.4.0 release, will be retained along with the Widget Library and SDK. > Administrative UIs will be added and there's also a desire to start > introducing some of the new unimplemented UI designs along the way, as these > have been through several rounds of usability testing and were presented to > the community at the summer Sakai/Jasig conference. > > The team has set itself a mid-December deadline--with intermediate deadlines > and sanity checks along the way--to produce working code that demonstrates > that the chosen path is viable. Attached is the September progress report > outlining the first 30 days of work along with our API doc and “alternative > scenarios” proposal. While the proposed architecture remains under > evaluation and could change, we believe that our recommendation to retire OAE > Nakamura in favor of a re-architected multi-tenant, cloud-ready platform > featuring purposefully designed REST APIs and services that are properly > instrumented and continually measured is the right approach to take. > > Kind regards, > Nicolaas and Anthony > > [1] http://nodejs.org/, http://cassandra.apache.org/ > [2] https://github.com/sakaiproject/Hilary > [3] https://github.com/sakaiproject/OAE-model-loader, > https://github.com/sakaiproject/node-oae-tsung, > https://github.com/sakaiproject/oae-nightly-stats, > https://github.com/sakaiproject/puppet-hilary > > <September Progress > Report.pdf><API.pdf>_______________________________________________ > oae-dev mailing list > oae-dev@collab.sakaiproject.org > http://collab.sakaiproject.org/mailman/listinfo/oae-dev _______________________________________________ oae-dev mailing list oae-dev@collab.sakaiproject.org http://collab.sakaiproject.org/mailman/listinfo/oae-dev
