Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points
On Tuesday 18 Nov 2008, Mark Easton wrote: We are charting observations from a range of sensors and contact inputs. The contact inputs are a special case as they can only have 1 or 0 values. But the sensors include temperature, current, voltage, light, humidity, vibration, motion, gps velocity, gps location, wind speed, wind direction etc. You could assing a color to each sensor, the 0/1 gives if it is drawn or not. Stack each sensor layer on top of each other, blend, and deliver a PNG to the client ? -- Tom Chiverton Helping to confidentially disseminate sticky cross-platform IPOs This email is sent for and on behalf of Halliwells LLP. Halliwells LLP is a limited liability partnership registered in England and Wales under registered number OC307980 whose registered office address is at Halliwells LLP, 3 Hardman Square, Spinningfields, Manchester, M3 3EB. A list of members is available for inspection at the registered office. Any reference to a partner in relation to Halliwells LLP means a member of Halliwells LLP. Regulated by The Solicitors Regulation Authority. CONFIDENTIALITY This email is intended only for the use of the addressee named above and may be confidential or legally privileged. If you are not the addressee you must not read it and must not use any information contained in nor copy it nor inform any person other than Halliwells LLP or the addressee of its existence or contents. If you have received this email in error please delete it and notify Halliwells LLP IT Department on 0870 365 2500. For more information about Halliwells LLP visit www.halliwells.com. -- Flexcoders Mailing List FAQ: http://groups.yahoo.com/group/flexcoders/files/flexcodersFAQ.txt Alternative FAQ location: https://share.acrobat.com/adc/document.do?docid=942dbdc8-e469-446f-b4cf-1e62079f6847 Search Archives: http://www.mail-archive.com/flexcoders%40yahoogroups.comYahoo! Groups Links * To visit your group on the web, go to: http://groups.yahoo.com/group/flexcoders/ * Your email settings: Individual Email | Traditional * To change settings online go to: http://groups.yahoo.com/group/flexcoders/join (Yahoo! ID required) * To change settings via email: mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] * To unsubscribe from this group, send an email to: [EMAIL PROTECTED] * Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points
What kind of charts are you displaying? I've had amazing success with scrapping the charting framework all together and simply drawing the chart programmatically. Yes, it's a bit more work, and yes it's only feasible if you don't really need a lot of the charting features that come in the framework. However, the performance difference is huge. The project I applied this on was for financial stock data, and using a custom drawn chart I mention above, along with the concept someone else mentioned by only showing the n-th datapoint interval depending on the overall range being displayed worked quite well. You're able to keep a consistent number of plot points (I've found 500-750 performs best) no matter what the overall range of data you're displaying. I say 500-750 performs best as I also had a horizontal axis slider which allowed for constantly updating data in the chart. Brendan On Tue, Nov 18, 2008 at 1:27 PM, jim.abbott45 [EMAIL PROTECTED]wrote: 50,000? LOL. ;-) On a more serious note, I have to agree with Fotis and Ricky that 50K data points is too many, both from a (Flash 9 VM at least) performance perspective and probably also from an Information Visualization perspective (unless, maybe, your users have 300 dpi monitors). There are several standard interaction design and information visualization techniques which may be useful to you . . . 1) Aggregation (as already suggested, take 'N' data points, average them, and then only display the average value). 2) Filtering (by sliders on an axis, by date/time, by structured or open-ended queries, ...) 3) Progressive rendering (sample the entire data set at intervals of 'N', render those points, then go back and get the data which is mid-point between the original points, add the new points to the curve, repeat until desired resolution obtained--or on user demand) 4) Non-linear zooming (like idea 1), but allow user to zoom in on a section of the curve, when they do, add the additional data points to the curve for the zoomed-in region) On a more pragmatic note, you certainly _can_ plot more than 2,000 points. For example, I've personally used Flex to create charts with over 14,000 points in them, so I know that at least that number is definitely *possible*. However, the overall rendering time was (as I recall) over 3 minutes long. More troubling was the fact that the entire Flex UI become very sluggish once a chart that large was displayed. I've seen good chart rendering speed ( ~= 2S ) and no UI sluggishness, up to about 1500 points. I've also been able to render up to 6,000 points, but it took about 10S for the rendering and the UI started to become sluggish. From your posting, it sounds like you're not seeing those rendering speeds. I'd recommend that you experiment with how you are using the charting objects/API. My experience was that I was able to see some very noticeable speed-ups simply be optimizing how I called the charting API (especially w.r.t. how I loaded the series data). In particular, if your charts are interactive, try to cache anything you can that doesn't necessarily change during an interaction (i.e., axis objects). Last thought: the new Flash 10 VM (Astro) has support for larger bitmaps and also can now do hardware-based rendering, apparently. I haven't tried either of those features myself, but using the v10 VM _might_ be another way to get a performance boost, etc. Good luck! Jim --- In flexcoders@yahoogroups.com flexcoders%40yahoogroups.com, Mark Easton [EMAIL PROTECTED] wrote: Hi, It appears that Flex Charts cannot handle generating charts with large DataSets. We tried with 50,000 data points and it thrashed away without producing a result after 6 minutes. It was able to plot 2,000 points in about 25 seconds. What is the recommended approach for creating charts from large data sets. The best I can think of is to write some code that will reduce the data set in size yet still provide enough data to represent the graph accurately. Thoughts? Ta. Mark -- Brendan Meutzner http://www.meutzner.com/blog/
RE: [flexcoders] Re: Creating a Chart With 50,000 Data Points
Hi Jim, Yes, LOL ... and it gets worse when we try to plot a graph for one year (about 500,000 observatons). We are plotting values received from sensors (eg temperature, vibration, humidity, gps location, gps velocity, wind speed, current, voltage, light). We take readings each minute. I have produced a dashboard which uses the following techniques: 1. Each sensor graph is in its own window ... so can be looking at 10 graphs at the same time for instance. 2. I average data to reduce the dataset size 3. I employ drill down 4. I have included a slider to look at reduced time frames The client rejected this solution. I did some tests in flex and found results similar to you Jim. When I tried to plot 40,000 points I gave up waiting for it after 6 minutes. 10,000 points took about 20 secs the first time (including reading from the local sqlite data store). The client and I have now agreed that: 1. If we have a large number of observations from a sensor and plot all of those on a graph over a period of time, then whatever optimisation we use to improve performance should produce an identical looking graph to the first. (I think that identical is probably to strong a word here) 2. We will add a confif paramter for each sensor such that the sensor readings can be changed to occur at multiples of 1 minute. For example, every 10 minutes. So, I will still end up with data sets to large for flex, and given that I want to display many graphs at once and will also be looking at representing many sensors on the one graph this contributes to the issue. So, I do need to sample the dataset somehow to make it work. But have to try to meet the goal expressed in point 1 above. I think a combination of your points 3 and 4 might be a good approached combined with some algorithm to ensure that my top level curve does not miss an max's and min's and as close as possible approximates a full detail curve. A few challenges there! Thanks again, Mark _ From: flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] On Behalf Of jim.abbott45 Sent: Wednesday, November 19, 2008 8:27 AM To: flexcoders@yahoogroups.com Subject: [flexcoders] Re: Creating a Chart With 50,000 Data Points 50,000? LOL. ;-) On a more serious note, I have to agree with Fotis and Ricky that 50K data points is too many, both from a (Flash 9 VM at least) performance perspective and probably also from an Information Visualization perspective (unless, maybe, your users have 300 dpi monitors). There are several standard interaction design and information visualization techniques which may be useful to you . . . 1) Aggregation (as already suggested, take 'N' data points, average them, and then only display the average value). 2) Filtering (by sliders on an axis, by date/time, by structured or open-ended queries, ...) 3) Progressive rendering (sample the entire data set at intervals of 'N', render those points, then go back and get the data which is mid-point between the original points, add the new points to the curve, repeat until desired resolution obtained--or on user demand) 4) Non-linear zooming (like idea 1), but allow user to zoom in on a section of the curve, when they do, add the additional data points to the curve for the zoomed-in region) On a more pragmatic note, you certainly _can_ plot more than 2,000 points. For example, I've personally used Flex to create charts with over 14,000 points in them, so I know that at least that number is definitely *possible*. However, the overall rendering time was (as I recall) over 3 minutes long. More troubling was the fact that the entire Flex UI become very sluggish once a chart that large was displayed. I've seen good chart rendering speed ( ~= 2S ) and no UI sluggishness, up to about 1500 points. I've also been able to render up to 6,000 points, but it took about 10S for the rendering and the UI started to become sluggish. From your posting, it sounds like you're not seeing those rendering speeds. I'd recommend that you experiment with how you are using the charting objects/API. My experience was that I was able to see some very noticeable speed-ups simply be optimizing how I called the charting API (especially w.r.t. how I loaded the series data). In particular, if your charts are interactive, try to cache anything you can that doesn't necessarily change during an interaction (i.e., axis objects). Last thought: the new Flash 10 VM (Astro) has support for larger bitmaps and also can now do hardware-based rendering, apparently. I haven't tried either of those features myself, but using the v10 VM _might_ be another way to get a performance boost, etc. Good luck! Jim --- In [EMAIL PROTECTED] mailto:flexcoders%40yahoogroups.com ups.com, Mark Easton [EMAIL PROTECTED] wrote: Hi, It appears that Flex Charts cannot handle generating charts with large DataSets. We tried with 50,000 data points and it thrashed away without producing a result after 6 minutes. It was able to plot
RE: [flexcoders] Re: Creating a Chart With 50,000 Data Points
Thanks Brendan, We are charting observations from a range of sensors and contact inputs. The contact inputs are a special case as they can only have 1 or 0 values. But the sensors include temperature, current, voltage, light, humidity, vibration, motion, gps velocity, gps location, wind speed, wind direction etc. The client does like the the charting features so he wants all of the flex charting cool stuff as well as the ability to see these charts. But that does not mean that I cant look at a combination of approaches. Its a good idea to look at coding a custom chart, and one I will look at closely. Yes I was thinking along the lines of having 2000 plot points. So in the case of a 3 month graph I would start with 120,000 points and need to reduce that to 2000. Cheers, Mark _ From: flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] On Behalf Of Brendan Meutzner Sent: Wednesday, November 19, 2008 8:51 AM To: flexcoders@yahoogroups.com Subject: Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points What kind of charts are you displaying? I've had amazing success with scrapping the charting framework all together and simply drawing the chart programmatically. Yes, it's a bit more work, and yes it's only feasible if you don't really need a lot of the charting features that come in the framework. However, the performance difference is huge. The project I applied this on was for financial stock data, and using a custom drawn chart I mention above, along with the concept someone else mentioned by only showing the n-th datapoint interval depending on the overall range being displayed worked quite well. You're able to keep a consistent number of plot points (I've found 500-750 performs best) no matter what the overall range of data you're displaying. I say 500-750 performs best as I also had a horizontal axis slider which allowed for constantly updating data in the chart. Brendan On Tue, Nov 18, 2008 at 1:27 PM, jim.abbott45 jim.abbott45@ mailto:[EMAIL PROTECTED] yahoo.com wrote: 50,000? LOL. ;-) On a more serious note, I have to agree with Fotis and Ricky that 50K data points is too many, both from a (Flash 9 VM at least) performance perspective and probably also from an Information Visualization perspective (unless, maybe, your users have 300 dpi monitors). There are several standard interaction design and information visualization techniques which may be useful to you . . . 1) Aggregation (as already suggested, take 'N' data points, average them, and then only display the average value). 2) Filtering (by sliders on an axis, by date/time, by structured or open-ended queries, ...) 3) Progressive rendering (sample the entire data set at intervals of 'N', render those points, then go back and get the data which is mid-point between the original points, add the new points to the curve, repeat until desired resolution obtained--or on user demand) 4) Non-linear zooming (like idea 1), but allow user to zoom in on a section of the curve, when they do, add the additional data points to the curve for the zoomed-in region) On a more pragmatic note, you certainly _can_ plot more than 2,000 points. For example, I've personally used Flex to create charts with over 14,000 points in them, so I know that at least that number is definitely *possible*. However, the overall rendering time was (as I recall) over 3 minutes long. More troubling was the fact that the entire Flex UI become very sluggish once a chart that large was displayed. I've seen good chart rendering speed ( ~= 2S ) and no UI sluggishness, up to about 1500 points. I've also been able to render up to 6,000 points, but it took about 10S for the rendering and the UI started to become sluggish. From your posting, it sounds like you're not seeing those rendering speeds. I'd recommend that you experiment with how you are using the charting objects/API. My experience was that I was able to see some very noticeable speed-ups simply be optimizing how I called the charting API (especially w.r.t. how I loaded the series data). In particular, if your charts are interactive, try to cache anything you can that doesn't necessarily change during an interaction (i.e., axis objects). Last thought: the new Flash 10 VM (Astro) has support for larger bitmaps and also can now do hardware-based rendering, apparently. I haven't tried either of those features myself, but using the v10 VM _might_ be another way to get a performance boost, etc. Good luck! Jim --- In [EMAIL PROTECTED] mailto:flexcoders%40yahoogroups.com ups.com, Mark Easton [EMAIL PROTECTED] wrote: Hi, It appears that Flex Charts cannot handle generating charts with large DataSets. We tried with 50,000 data points and it thrashed away without producing a result after 6 minutes. It was able to plot 2,000 points in about 25 seconds. What is the recommended approach for creating charts from large data sets. The best I can think of is to write some code
Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points
I'd look into some intermediary code to reduce your dataset, written in something with threads (my first choice would be Java, but that's me). It sounds like you're working on some interesting stuff Mark :) -Josh On Wed, Nov 19, 2008 at 6:15 AM, Mark Easton [EMAIL PROTECTED] wrote: Thanks Brendan, We are charting observations from a range of sensors and contact inputs. The contact inputs are a special case as they can only have 1 or 0 values. But the sensors include temperature, current, voltage, light, humidity, vibration, motion, gps velocity, gps location, wind speed, wind direction etc. The client does like the the charting features so he wants all of the flex charting cool stuff as well as the ability to see these charts. But that does not mean that I cant look at a combination of approaches. Its a good idea to look at coding a custom chart, and one I will look at closely. Yes I was thinking along the lines of having 2000 plot points. So in the case of a 3 month graph I would start with 120,000 points and need to reduce that to 2000. Cheers, Mark -- *From:* flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] *On Behalf Of *Brendan Meutzner *Sent:* Wednesday, November 19, 2008 8:51 AM *To:* flexcoders@yahoogroups.com *Subject:* Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points What kind of charts are you displaying? I've had amazing success with scrapping the charting framework all together and simply drawing the chart programmatically. Yes, it's a bit more work, and yes it's only feasible if you don't really need a lot of the charting features that come in the framework. However, the performance difference is huge. The project I applied this on was for financial stock data, and using a custom drawn chart I mention above, along with the concept someone else mentioned by only showing the n-th datapoint interval depending on the overall range being displayed worked quite well. You're able to keep a consistent number of plot points (I've found 500-750 performs best) no matter what the overall range of data you're displaying. I say 500-750 performs best as I also had a horizontal axis slider which allowed for constantly updating data in the chart. Brendan On Tue, Nov 18, 2008 at 1:27 PM, jim.abbott45 [EMAIL PROTECTED]wrote: 50,000? LOL. ;-) On a more serious note, I have to agree with Fotis and Ricky that 50K data points is too many, both from a (Flash 9 VM at least) performance perspective and probably also from an Information Visualization perspective (unless, maybe, your users have 300 dpi monitors). There are several standard interaction design and information visualization techniques which may be useful to you . . . 1) Aggregation (as already suggested, take 'N' data points, average them, and then only display the average value). 2) Filtering (by sliders on an axis, by date/time, by structured or open-ended queries, ...) 3) Progressive rendering (sample the entire data set at intervals of 'N', render those points, then go back and get the data which is mid-point between the original points, add the new points to the curve, repeat until desired resolution obtained--or on user demand) 4) Non-linear zooming (like idea 1), but allow user to zoom in on a section of the curve, when they do, add the additional data points to the curve for the zoomed-in region) On a more pragmatic note, you certainly _can_ plot more than 2,000 points. For example, I've personally used Flex to create charts with over 14,000 points in them, so I know that at least that number is definitely *possible*. However, the overall rendering time was (as I recall) over 3 minutes long. More troubling was the fact that the entire Flex UI become very sluggish once a chart that large was displayed. I've seen good chart rendering speed ( ~= 2S ) and no UI sluggishness, up to about 1500 points. I've also been able to render up to 6,000 points, but it took about 10S for the rendering and the UI started to become sluggish. From your posting, it sounds like you're not seeing those rendering speeds. I'd recommend that you experiment with how you are using the charting objects/API. My experience was that I was able to see some very noticeable speed-ups simply be optimizing how I called the charting API (especially w.r.t. how I loaded the series data). In particular, if your charts are interactive, try to cache anything you can that doesn't necessarily change during an interaction (i.e., axis objects). Last thought: the new Flash 10 VM (Astro) has support for larger bitmaps and also can now do hardware-based rendering, apparently. I haven't tried either of those features myself, but using the v10 VM _might_ be another way to get a performance boost, etc. Good luck! Jim --- In flexcoders@yahoogroups.com flexcoders%40yahoogroups.com, Mark Easton [EMAIL PROTECTED] wrote: Hi, It appears
Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points
You can always try a Java Applet. But this suggestion might be off topic! On Wed, Nov 19, 2008 at 12:04 AM, Mark Easton [EMAIL PROTECTED]wrote: I'd be using C++. :) I already have a C++ application running on our gateway which deals with grabbing data from the stations. It then passes that to a parser utility I wrote so that would be a good place to reduce the dataset ready for a progressive display on a flex chart. Yep - a very interesting application! -- *From:* flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] *On Behalf Of *Josh McDonald *Sent:* Wednesday, November 19, 2008 11:34 AM *To:* flexcoders@yahoogroups.com *Subject:* Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points I'd look into some intermediary code to reduce your dataset, written in something with threads (my first choice would be Java, but that's me). It sounds like you're working on some interesting stuff Mark :) -Josh On Wed, Nov 19, 2008 at 6:15 AM, Mark Easton [EMAIL PROTECTED]wrote: Thanks Brendan, We are charting observations from a range of sensors and contact inputs. The contact inputs are a special case as they can only have 1 or 0 values. But the sensors include temperature, current, voltage, light, humidity, vibration, motion, gps velocity, gps location, wind speed, wind direction etc. The client does like the the charting features so he wants all of the flex charting cool stuff as well as the ability to see these charts. But that does not mean that I cant look at a combination of approaches. Its a good idea to look at coding a custom chart, and one I will look at closely. Yes I was thinking along the lines of having 2000 plot points. So in the case of a 3 month graph I would start with 120,000 points and need to reduce that to 2000. Cheers, Mark -- *From:* flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] *On Behalf Of *Brendan Meutzner *Sent:* Wednesday, November 19, 2008 8:51 AM *To:* flexcoders@yahoogroups.com *Subject:* Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points What kind of charts are you displaying? I've had amazing success with scrapping the charting framework all together and simply drawing the chart programmatically. Yes, it's a bit more work, and yes it's only feasible if you don't really need a lot of the charting features that come in the framework. However, the performance difference is huge. The project I applied this on was for financial stock data, and using a custom drawn chart I mention above, along with the concept someone else mentioned by only showing the n-th datapoint interval depending on the overall range being displayed worked quite well. You're able to keep a consistent number of plot points (I've found 500-750 performs best) no matter what the overall range of data you're displaying. I say 500-750 performs best as I also had a horizontal axis slider which allowed for constantly updating data in the chart. Brendan On Tue, Nov 18, 2008 at 1:27 PM, jim.abbott45 [EMAIL PROTECTED]wrote: 50,000? LOL. ;-) On a more serious note, I have to agree with Fotis and Ricky that 50K data points is too many, both from a (Flash 9 VM at least) performance perspective and probably also from an Information Visualization perspective (unless, maybe, your users have 300 dpi monitors). There are several standard interaction design and information visualization techniques which may be useful to you . . . 1) Aggregation (as already suggested, take 'N' data points, average them, and then only display the average value). 2) Filtering (by sliders on an axis, by date/time, by structured or open-ended queries, ...) 3) Progressive rendering (sample the entire data set at intervals of 'N', render those points, then go back and get the data which is mid-point between the original points, add the new points to the curve, repeat until desired resolution obtained--or on user demand) 4) Non-linear zooming (like idea 1), but allow user to zoom in on a section of the curve, when they do, add the additional data points to the curve for the zoomed-in region) On a more pragmatic note, you certainly _can_ plot more than 2,000 points. For example, I've personally used Flex to create charts with over 14,000 points in them, so I know that at least that number is definitely *possible*. However, the overall rendering time was (as I recall) over 3 minutes long. More troubling was the fact that the entire Flex UI become very sluggish once a chart that large was displayed. I've seen good chart rendering speed ( ~= 2S ) and no UI sluggishness, up to about 1500 points. I've also been able to render up to 6,000 points, but it took about 10S for the rendering and the UI started to become sluggish. From your posting, it sounds like you're not seeing those rendering speeds. I'd recommend that you experiment with how you are using
RE: [flexcoders] Re: Creating a Chart With 50,000 Data Points
A little bit rusty on Java right now - did some J2ME last year but not much since. _ From: flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] On Behalf Of Romain de Wolff Sent: Wednesday, November 19, 2008 12:06 PM To: flexcoders@yahoogroups.com Subject: Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points You can always try a Java Applet. But this suggestion might be off topic! On Wed, Nov 19, 2008 at 12:04 AM, Mark Easton [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] co.nz wrote: I'd be using C++. :) I already have a C++ application running on our gateway which deals with grabbing data from the stations. It then passes that to a parser utility I wrote so that would be a good place to reduce the dataset ready for a progressive display on a flex chart. Yep - a very interesting application! _ From: [EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com [mailto:[EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com] On Behalf Of Josh McDonald Sent: Wednesday, November 19, 2008 11:34 AM To: [EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com Subject: Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points I'd look into some intermediary code to reduce your dataset, written in something with threads (my first choice would be Java, but that's me). It sounds like you're working on some interesting stuff Mark :) -Josh On Wed, Nov 19, 2008 at 6:15 AM, Mark Easton [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] co.nz wrote: Thanks Brendan, We are charting observations from a range of sensors and contact inputs. The contact inputs are a special case as they can only have 1 or 0 values. But the sensors include temperature, current, voltage, light, humidity, vibration, motion, gps velocity, gps location, wind speed, wind direction etc. The client does like the the charting features so he wants all of the flex charting cool stuff as well as the ability to see these charts. But that does not mean that I cant look at a combination of approaches. Its a good idea to look at coding a custom chart, and one I will look at closely. Yes I was thinking along the lines of having 2000 plot points. So in the case of a 3 month graph I would start with 120,000 points and need to reduce that to 2000. Cheers, Mark _ From: [EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com [mailto:[EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com] On Behalf Of Brendan Meutzner Sent: Wednesday, November 19, 2008 8:51 AM To: [EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com Subject: Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points What kind of charts are you displaying? I've had amazing success with scrapping the charting framework all together and simply drawing the chart programmatically. Yes, it's a bit more work, and yes it's only feasible if you don't really need a lot of the charting features that come in the framework. However, the performance difference is huge. The project I applied this on was for financial stock data, and using a custom drawn chart I mention above, along with the concept someone else mentioned by only showing the n-th datapoint interval depending on the overall range being displayed worked quite well. You're able to keep a consistent number of plot points (I've found 500-750 performs best) no matter what the overall range of data you're displaying. I say 500-750 performs best as I also had a horizontal axis slider which allowed for constantly updating data in the chart. Brendan On Tue, Nov 18, 2008 at 1:27 PM, jim.abbott45 jim.abbott45@ mailto:[EMAIL PROTECTED] yahoo.com wrote: 50,000? LOL. ;-) On a more serious note, I have to agree with Fotis and Ricky that 50K data points is too many, both from a (Flash 9 VM at least) performance perspective and probably also from an Information Visualization perspective (unless, maybe, your users have 300 dpi monitors). There are several standard interaction design and information visualization techniques which may be useful to you . . . 1) Aggregation (as already suggested, take 'N' data points, average them, and then only display the average value). 2) Filtering (by sliders on an axis, by date/time, by structured or open-ended queries, ...) 3) Progressive rendering (sample the entire data set at intervals of 'N', render those points, then go back and get the data which is mid-point between the original points, add the new points to the curve, repeat until desired resolution obtained--or on user demand) 4) Non-linear zooming (like idea 1), but allow user to zoom in on a section of the curve, when they do, add the additional data points to the curve for the zoomed-in region) On a more pragmatic note, you certainly _can_ plot more than 2,000 points. For example, I've personally used Flex to create charts with over 14,000 points in them, so I know that at least that number is definitely *possible*. However, the overall
RE: [flexcoders] Re: Creating a Chart With 50,000 Data Points
I'd be using C++. :) I already have a C++ application running on our gateway which deals with grabbing data from the stations. It then passes that to a parser utility I wrote so that would be a good place to reduce the dataset ready for a progressive display on a flex chart. Yep - a very interesting application! _ From: flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] On Behalf Of Josh McDonald Sent: Wednesday, November 19, 2008 11:34 AM To: flexcoders@yahoogroups.com Subject: Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points I'd look into some intermediary code to reduce your dataset, written in something with threads (my first choice would be Java, but that's me). It sounds like you're working on some interesting stuff Mark :) -Josh On Wed, Nov 19, 2008 at 6:15 AM, Mark Easton [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] co.nz wrote: Thanks Brendan, We are charting observations from a range of sensors and contact inputs. The contact inputs are a special case as they can only have 1 or 0 values. But the sensors include temperature, current, voltage, light, humidity, vibration, motion, gps velocity, gps location, wind speed, wind direction etc. The client does like the the charting features so he wants all of the flex charting cool stuff as well as the ability to see these charts. But that does not mean that I cant look at a combination of approaches. Its a good idea to look at coding a custom chart, and one I will look at closely. Yes I was thinking along the lines of having 2000 plot points. So in the case of a 3 month graph I would start with 120,000 points and need to reduce that to 2000. Cheers, Mark _ From: [EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com [mailto:[EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com] On Behalf Of Brendan Meutzner Sent: Wednesday, November 19, 2008 8:51 AM To: [EMAIL PROTECTED] mailto:flexcoders@yahoogroups.com ups.com Subject: Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points What kind of charts are you displaying? I've had amazing success with scrapping the charting framework all together and simply drawing the chart programmatically. Yes, it's a bit more work, and yes it's only feasible if you don't really need a lot of the charting features that come in the framework. However, the performance difference is huge. The project I applied this on was for financial stock data, and using a custom drawn chart I mention above, along with the concept someone else mentioned by only showing the n-th datapoint interval depending on the overall range being displayed worked quite well. You're able to keep a consistent number of plot points (I've found 500-750 performs best) no matter what the overall range of data you're displaying. I say 500-750 performs best as I also had a horizontal axis slider which allowed for constantly updating data in the chart. Brendan On Tue, Nov 18, 2008 at 1:27 PM, jim.abbott45 jim.abbott45@ mailto:[EMAIL PROTECTED] yahoo.com wrote: 50,000? LOL. ;-) On a more serious note, I have to agree with Fotis and Ricky that 50K data points is too many, both from a (Flash 9 VM at least) performance perspective and probably also from an Information Visualization perspective (unless, maybe, your users have 300 dpi monitors). There are several standard interaction design and information visualization techniques which may be useful to you . . . 1) Aggregation (as already suggested, take 'N' data points, average them, and then only display the average value). 2) Filtering (by sliders on an axis, by date/time, by structured or open-ended queries, ...) 3) Progressive rendering (sample the entire data set at intervals of 'N', render those points, then go back and get the data which is mid-point between the original points, add the new points to the curve, repeat until desired resolution obtained--or on user demand) 4) Non-linear zooming (like idea 1), but allow user to zoom in on a section of the curve, when they do, add the additional data points to the curve for the zoomed-in region) On a more pragmatic note, you certainly _can_ plot more than 2,000 points. For example, I've personally used Flex to create charts with over 14,000 points in them, so I know that at least that number is definitely *possible*. However, the overall rendering time was (as I recall) over 3 minutes long. More troubling was the fact that the entire Flex UI become very sluggish once a chart that large was displayed. I've seen good chart rendering speed ( ~= 2S ) and no UI sluggishness, up to about 1500 points. I've also been able to render up to 6,000 points, but it took about 10S for the rendering and the UI started to become sluggish. From your posting, it sounds like you're not seeing those rendering speeds. I'd recommend that you experiment with how you are using the charting objects/API. My experience was that I was able to see some very noticeable speed-ups simply
Re: [flexcoders] Re: Creating a Chart With 50,000 Data Points
J2ME... My condolences ;-) On Wed, Nov 19, 2008 at 9:08 AM, Mark Easton [EMAIL PROTECTED] wrote: A little bit rusty on Java right now - did some J2ME last year but not much since. -- Therefore, send not to know For whom the bell tolls. It tolls for thee. Like the cut of my jib? Check out my Flex blog! :: Josh 'G-Funk' McDonald :: 0437 221 380 :: [EMAIL PROTECTED] :: http://flex.joshmcdonald.info/ :: http://twitter.com/sophistifunk
