hi Raja, yes... it gets intersting... and will become more so, lets keep the intelligent discussions going, without getting too serious ;-)
As for tools and platforms.... numpy, scipy, rpy these can be both client side or backend tools. They follow Unix traditions, they can be entirely script driven - write a number crunching app (in Python of course) pass it some inputs, say from a file, the output goes into another file, apply plotting routines to the files, generate graphs, export graphs as PNGs serve it out on the web. So except for the last part, the rest can take place on the server side, maybe with a webUI to control which programe to apply to which dataset, etc... However, if the user want to manipulate the data live, ie: select a range off the graph axis and apply a transformation to it... then you need the sciPy stuff on your local PC, and the UI has to be alto better than just Web, probably GTK, QT or Cocoa. Dunno if JS can do that? Ah frameworks... big controversial subject! Let me throw in my 2 cents worth. When I started building my LIS (Lab Info System) I too was deciding on a framework, and its not easy simply becos Python has too many choices, look at Python.org: software, web-frameworks and you see >20/30 frameworks, yikes! My reasoning was thus, my problem domain was not well defined, after much googling, there was no clear definition of what a LIS should be, should do. Also even the definitions I find are for developed countries where, procedures, level of skills of people working in the Labs etc... is certainly different from ours. Good thing the doctor I work with, a hemotologist is a domain expert, having worked in a big Lab in US for a number of years, and he was able to define, what he wanted, and even better, what was his vision. Thus I needed flexibility, I may change things 1/2 way thru, I did not want a framework that forced/committed me to a certain way of doing things which was difficult to change later. I was also motivated by another FOSS project, Sahana, a SW for disaster management, that was developed immd. after the last Tsunami, and it taught me that medical systems/apps need to be run in the field, not just in the data-centre (like in a medic tent, after phone lines and networks are knocked out, with limited power supply....). So part of my requirement, is that the LIS had to be 'lightweight' enogh to run in the field, on an average laptop. So in this situation, I ruled out heavy entreprise type frameworks, yet the app had to be scalable... Also, since I attempted to write my own framework earlier(with Python its not that difficult..) I knew exactly what I wanted, and what to avoid. I did not want a heavy framework, loaded with a lot of features that did things 'their' way. Its OK if what you want to do coincided, but once you diverge form it, there's a lot of pain. Also with big frameworks, libraries etc.. you spend as much time learning how they function so you can modify it; than if you were to develop some of these features yourself. Access and user management for example, if you wanted fine-graine control for user access to data and functionalities... and you wanted an hierachical organization of users, there would be a problem with most std. frameworks, you wld have to start modifying things.. I wanted a lightwieght framework that provided the bare skeleton and I can the build my feature set around them, my way. The trade-off of course is that you have to build more things yourself, whereas the big frameworks alread has these... if it suits yr use. And I wanted a good, powerful, flexible API with RDBMS - one that made the DB 'transparent' and I can switch DB if need be. In the end I settled on Twisted, a lightwieght asynchronous framework (which means its scalable), its feature rich, not only has web, but SMTP, POP, FTP, etc... a whole bunch of other servers, a very powerful tmeplating engine, good session management, even provides direct JS to server HTTP connections for AJAX stuff... all in a package that's about 7Mb, compared to 30+ for Django, Zope/Plone. Its also got a low memory foorprint, I can run multiple instances of the server on 1 CPU w/o problem. Another Python framework I'm looking at is seriously, is Nagare, its based on Stackless Python, a highly concurrent Python that has continuations built in, ala Scala and is very scalable. It has a completely different paradigm for building web apps, where each 'widget' on a web page is its own micro-thread, instead of a big monolithic structure. Weird but intereting, So it depends is your app domain, is it well defined, can you specify it precisely, do you need flexibilty, change it as you go along, adding features, plug-ins.... etc... BTW what is "etl" ? wrt SnapLogic. You'll find that developing in Python gives you power and productivity, thats why Google uses it... As for yr analytics tool, have you heard of GapMinder, Hans Rosling and the fact that Google bought it. If you hav'nt this will interst you... http://www.ted.com/speakers/hans_rosling.html Gapminder analyzes economic data and present it in a very impressive way, Hans Rosling is a Health Economist, I think, but can be used for any data... its right up there with Freakonomics and stuff... for coolness level, in a very geek/methematical way. As for VC's knocking on yr door with buckets of cash, if you build this magical SW, I don't thnk it will happen in Malaysia. I'm cynical after being on this path for too long... A few items of 'bad' news... MDEC funding has dried up, finished off their budget. And I have been to a few TeAM (Technopreneurs Asscoc. of Malaysia) meetings and met/listened VC's there & elsewhere... basically Malaysian VCs are interested to talk to you only AFTER you have a product AND CUSTOMERS! If I have these, why do I need VC's, just get a bank loan! So the morale of the story is... Malaysian VC's are more like bankers than VC, they don't like risk! Also Msian VC's have little clue on technology, they can only look backwards, for the next Google (that will compete withh Google.... duh?), but they cannot see forward! the "Blue ocean" is quite beyond most of them, those I've met anyway. Also Raja, you have some interesting ideas, and I would like to bounce some ideas off you too, perhaps we should meet up for a more in-depth discussion. Where and when shall we meet? On Fri, Apr 23, 2010 at 11:17 PM, Raja Iskandar Shah <[email protected]> wrote: > wahlau ..... suddenly this has become interesting > > quick replies > > azrul, i am looking for something other than java / point taken that scala > is not java ---- hopefully, this will not make me your immortal enemy > > amran, as boh rightly pointed out bi is oversold - now even crystal reports > is considered bi (b**ls**t). that's why i have labelled our product > analytics. unfortunately i googled for php etl and php modelling - but there > are none. > > sweemeng, thanks. from the scrapy site i found this http://data.gov.uk/ . > besides web scrapping, we also need file scrapping. for now we web scrape > using php, and file scrape using openoffice and java etl. > > chow, you read my thots correctly. yes, we are looking for modelling. > > boh, huge thanks ;-) yes, i am looking for mathematical and statistical > modelling that can handle huge arrays . besides scipy, numpy, i have also > found rpy (for statistics) > > also found an etl tool in python called snaplogic. > > next question, numpy, scipy and snaplogic are all client / desktop > applications. would django be a good web framework as a web ui to work with > numpy, scipy and snaplogic ? any good ide for python (code completion, > syntax highlighting, user defined functions / libraries, git) ? > > and finally, if i said i wanted to develop this on python because google > uses python... will capitalist be rushing to my front door offering wads of > dosh (meaning lots and lots of money ?) > > // freakonomics and wikinomics inspired some of the stuff i am thinking of > doing > > > > > On Fri, Apr 23, 2010 at 4:33 PM, Boh Yap <[email protected]> wrote: >> >> hi >> >> no offense to you or anyone else, but I can't help but reply.... >> >> BI is being made to be such a "big deal", its nothing more than a term >> being dreamt up by marketeers to sell something that economists have >> been doing for a long time: applying mathematical techniques to data, >> except that this data set is 'local' to a business, as opposed to a >> whole country. Its nothing more than analysing data and presenting it >> graphically, thats all. In the old days it was called DSS, or EIS >> (Decision Support System, Executive Info System). But the marketeers >> had to re-clothe the emperor each time... ;-) >> >> Its purpose was to present analysis and data to management types whose >> knowledge of maths is only mediocre (high-school calculus is not a >> prerequisitr to do a Business Degree...). >> >> And hacking data is hacking data, the tools are the same, in science >> or in biz, and scientists and mathematicians do it better than the >> biz types. >> >> Scientists/mathematicians (and often economists) look for truth in the >> numbers, biz. types look look for numbers to present their version of >> truth! (case in point: the recent wall street meltdown!) >> >> --apologies~~ the above is my attempt at humour, satire and sarcsm ;-) >> read the book "Freakonomics" - by SD Levit... >> >> > -- > Join Open Source Developers Club Malaysia http://www.osdc.my/ > > Facebook Fan page > > http://www.facebook.com/group.php?gid=98685301577 > > http://www.facebook.com/OSDC.my > > You received this message because you are subscribed to the Google > > Groups "OSDC.my Mailing List" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/osdcmy-list?hl=en -- #------- regds, Boh Heong, Yap -- Join Open Source Developers Club Malaysia http://www.osdc.my/ Facebook Fan page http://www.facebook.com/group.php?gid=98685301577 http://www.facebook.com/OSDC.my You received this message because you are subscribed to the Google Groups "OSDC.my Mailing List" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/osdcmy-list?hl=en
