Re: [O] Exploring data that is in org-mode format
jw.he...@gmail.com writes: For simple exploration, you might have a look at ggobi? [1] It allows you to do some really quick/easy exploration by plotting and being able to check which variables to use for X and Y, coloring, filtering, changing plot type, and so on. There's an R package which allows you to call ggobi on an R data object, which you could easily create with babel and your existing org tables with the #+name option for the table and :var specification in the babel block header. I'd also highly recommend taking a look at shiny via R-Studio.[2] Not sure if you can call it from Org-mode, but even if you can't... not *everything* has to be done with Org. You could use the file to do some data munging/summarization/etc., save it as a new data set (.csv or similar), and then read that into Shiny. It could be *awesome* for something like this. I only recently started playing with it but it's just fantastic and would make for the ability to subset, change scales/time ranges, and much more in an interactive web app. Thank you for these suggestions, they will definitely keep me occupied for a while. Alan
[O] Exploring data that is in org-mode format
Hello, This question is slightly off-topic, but it may be of interest to people who have a lot of data entered in org-mode. The short version: what tools are available to explore data, typically stored in org-mode tables? The long version: I've tried an interesting website (https://tictrac.com/) whose goal is to gain some insight about ourselves by exploring some data we collect (think quantified self). I'm not happy with this site for three reasons: - I need to send it the data; - it focuses on health / activity data whereas there is much more that interests me (I for instance have weekly records of natural gas use in my gas-heated house and daily record of temperature average outside which I would love to compare); - it won't let you input arbitrary data (I asked about importing a CSV of my daily coffee consumption, they answered they require an external service to integrate the data). So I collect all this data because it's something I enjoy doing, and I would really like to explore it, from the comfortable position of my own computer. All of this data is in org-mode tables (or can be easily converted to org-mode table). Hence my questions: are there tools you would recommend? I'm not afraid of programming (I suspect an answer will be 'R'), but I would like pointers to tutorials to do these kind of things. The kind of things I would like to do are: - extract weekly or monthly tallies or estimation from data collected at irregular intervals; - compare data sources against each other; - estimate future trends based on past data (how much will my gas bill be?); - display the result in some kind of dashboard. Thanks a lot, Alan
Re: [O] Exploring data that is in org-mode format
Dnia 2013-09-27, o godz. 10:18:15 Alan Schmitt alan.schm...@polytechnique.org napisał(a): Hello, This question is slightly off-topic, but it may be of interest to people who have a lot of data entered in org-mode. The short version: what tools are available to explore data, typically stored in org-mode tables? The long version: I've tried an interesting website (https://tictrac.com/) whose goal is to gain some insight about ourselves by exploring some data we collect (think quantified self). I'm not happy with this site for three reasons: - I need to send it the data; - it focuses on health / activity data whereas there is much more that interests me (I for instance have weekly records of natural gas use in my gas-heated house and daily record of temperature average outside which I would love to compare); - it won't let you input arbitrary data (I asked about importing a CSV of my daily coffee consumption, they answered they require an external service to integrate the data). So I collect all this data because it's something I enjoy doing, and I would really like to explore it, from the comfortable position of my own computer. All of this data is in org-mode tables (or can be easily converted to org-mode table). Hence my questions: are there tools you would recommend? I'm not afraid of programming (I suspect an answer will be 'R'), but I would like pointers to tutorials to do these kind of things. The kind of things I would like to do are: - extract weekly or monthly tallies or estimation from data collected at irregular intervals; - compare data sources against each other; - estimate future trends based on past data (how much will my gas bill be?); - display the result in some kind of dashboard. Thanks a lot, Interesting question. And although it is probably of no use for me, I'd love to see an Emacs-based tool to do that... Alan Best, -- Marcin Borkowski http://octd.wmi.amu.edu.pl/en/Marcin_Borkowski Adam Mickiewicz University
Re: [O] Exploring data that is in org-mode format
* Alan Schmitt alan.schm...@polytechnique.org wrote: Hello, Hi Alan! The short version: what tools are available to explore data, typically stored in org-mode tables? Great question! I did some very basic R-scripts to derive boxplots by myself. The long version: I've tried an interesting website (https://tictrac.com/) whose goal is to gain some insight about ourselves by exploring some data we collect (think quantified self). I'm not happy with this site for three reasons: - I need to send it the data; I dislike cloud-based services which are not under my control. - it focuses on health / activity data whereas there is much more that interests me (I for instance have weekly records of natural gas use in my gas-heated house and daily record of temperature average outside which I would love to compare); Absolutely! - it won't let you input arbitrary data (I asked about importing a CSV of my daily coffee consumption, they answered they require an external service to integrate the data). I understand. So I collect all this data because it's something I enjoy doing, and I would really like to explore it, from the comfortable position of my own computer. All of this data is in org-mode tables (or can be easily converted to org-mode table). Hence my questions: are there tools you would recommend? I'm not afraid of programming (I suspect an answer will be 'R'), but I would like pointers to tutorials to do these kind of things. The kind of things I would like to do are: - extract weekly or monthly tallies or estimation from data collected at irregular intervals; - compare data sources against each other; - estimate future trends based on past data (how much will my gas bill be?); - display the result in some kind of dashboard. I collect much data about myself using Memacs (see signature). However, I did not derive anything else but some boxplots from it (visualizing time-differences between periodic events). Something generic which I can feed Org-mode files (most preferably in Memacs-like Org-mode format) and derive some standard graphics (where I pick those that make sense) would be very great! -- mail|git|SVN|photos|postings|SMS|phonecalls|RSS|CSV|XML to Org-mode: get Memacs from https://github.com/novoid/Memacs https://github.com/novoid/extract_pdf_annotations_to_orgmode + more on github
Re: [O] Exploring data that is in org-mode format
Alan Schmitt alan.schm...@polytechnique.org writes: Hello, This question is slightly off-topic, but it may be of interest to people who have a lot of data entered in org-mode. The short version: what tools are available to explore data, typically stored in org-mode tables? The long version: I've tried an interesting website (https://tictrac.com/) whose goal is to gain some insight about ourselves by exploring some data we collect (think quantified self). I'm not happy with this site for three reasons: - I need to send it the data; - it focuses on health / activity data whereas there is much more that interests me (I for instance have weekly records of natural gas use in my gas-heated house and daily record of temperature average outside which I would love to compare); - it won't let you input arbitrary data (I asked about importing a CSV of my daily coffee consumption, they answered they require an external service to integrate the data). So I collect all this data because it's something I enjoy doing, and I would really like to explore it, from the comfortable position of my own computer. All of this data is in org-mode tables (or can be easily converted to org-mode table). Hence my questions: are there tools you would recommend? I'm not afraid of programming (I suspect an answer will be 'R'), but I would like pointers to tutorials to do these kind of things. The kind of things I would like to do are: - extract weekly or monthly tallies or estimation from data collected at irregular intervals; - compare data sources against each other; - estimate future trends based on past data (how much will my gas bill be?); - display the result in some kind of dashboard. Not org-related and not even emacs-related (sorry Marcin!) but applicable to the question: Apart from R and Matlab, there is also ... Python: I'm currently reading a very nice book that uses Python, Numpy, Pandas and Matplotlib for data exploration. It is called Python for Data Analysis, by Wes McKinney (the original developer of Pandas). I'm about a third of the way through it and I can recommend it. You can find a link to the book at the Pandas site: http://pandas.pydata.org/ Just in case the question arises: no, I'm not a paid endorser - just a satisfied customer :-) -- Nick
Re: [O] Exploring data that is in org-mode format
On Fri, Sep 27, 2013 at 3:18 AM, Alan Schmitt alan.schm...@polytechnique.org wrote: Hello, This question is slightly off-topic, but it may be of interest to people who have a lot of data entered in org-mode. The short version: what tools are available to explore data, typically stored in org-mode tables? [snip] So I collect all this data because it's something I enjoy doing, and I would really like to explore it, from the comfortable position of my own computer. All of this data is in org-mode tables (or can be easily converted to org-mode table). Hence my questions: are there tools you would recommend? I'm not afraid of programming (I suspect an answer will be 'R'), but I would like pointers to tutorials to do these kind of things. The kind of things I would like to do are: - extract weekly or monthly tallies or estimation from data collected at irregular intervals; - compare data sources against each other; - estimate future trends based on past data (how much will my gas bill be?); - display the result in some kind of dashboard. For simple exploration, you might have a look at ggobi? [1] It allows you to do some really quick/easy exploration by plotting and being able to check which variables to use for X and Y, coloring, filtering, changing plot type, and so on. There's an R package which allows you to call ggobi on an R data object, which you could easily create with babel and your existing org tables with the #+name option for the table and :var specification in the babel block header. I'd also highly recommend taking a look at shiny via R-Studio.[2] Not sure if you can call it from Org-mode, but even if you can't... not *everything* has to be done with Org. You could use the file to do some data munging/summarization/etc., save it as a new data set (.csv or similar), and then read that into Shiny. It could be *awesome* for something like this. I only recently started playing with it but it's just fantastic and would make for the ability to subset, change scales/time ranges, and much more in an interactive web app. I applied for a public server account with RStudio and got it so that as I learn and do more, I can make them public. As a result of an SO question, I repaid the answerer by creating an app from his reply: - Post: http://stackoverflow.com/questions/17958730/faceting-a-set-of-contour-plots-in-ggplot-r - Shiny app: http://spark.rstudio.com/jwhendy/interactive-contour/ Good luck! John [1] http://www.ggobi.org/ [2] http://www.rstudio.com/shiny/ Thanks a lot, Alan
