Re: [flexcoders] Re: Flex grid to serve 100K-1mil rows
On Monday 12 Nov 2007, letterpigeon wrote: will have to go in and change the broker code 1 by 1 that could take them hours. Therefore, the ability for our users to perform mass actions on the dataset (either client side or server side) would be a huge win for us. So there's no need to wait for the action on a set of the trades to actually complete ? You can just add the change to a list of actions to be done, and batch process it when the user is done. The client could show a smaller subset 'preview' of their changes. -- Tom Chiverton Helping to autoschediastically e-enable enterprise-class communities on: http://thefalken.livejournal.com This email is sent for and on behalf of Halliwells LLP. Halliwells LLP is a limited liability partnership registered in England and Wales under registered number OC307980 whose registered office address is at St James's Court Brown Street Manchester M2 2JF. A list of members is available for inspection at the registered office. Any reference to a partner in relation to Halliwells LLP means a member of Halliwells LLP. Regulated by The Solicitors Regulation Authority. CONFIDENTIALITY This email is intended only for the use of the addressee named above and may be confidential or legally privileged. If you are not the addressee you must not read it and must not use any information contained in nor copy it nor inform any person other than Halliwells LLP or the addressee of its existence or contents. If you have received this email in error please delete it and notify Halliwells LLP IT Department on 0870 365 2500. For more information about Halliwells LLP visit www.halliwells.com. -- Flexcoders Mailing List FAQ: http://groups.yahoo.com/group/flexcoders/files/flexcodersFAQ.txt Search Archives: http://www.mail-archive.com/flexcoders%40yahoogroups.com Yahoo! Groups Links * To visit your group on the web, go to: http://groups.yahoo.com/group/flexcoders/ * Your email settings: Individual Email | Traditional * To change settings online go to: http://groups.yahoo.com/group/flexcoders/join (Yahoo! ID required) * To change settings via email: mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] * To unsubscribe from this group, send an email to: [EMAIL PROTECTED] * Your use of Yahoo! Groups is subject to: http://docs.yahoo.com/info/terms/
[flexcoders] Re: Flex grid to serve 100K-1mil rows
Hi William, I've thought of this idea, and actually tested the capacity of the client. I loaded up dummy data on the client side and it can easily take up to 1mil records (though filtering could take a few seconds). The problem is how do I send that much of data from the server side. I'm using openamf to communicate to a java servlet, the servlet goes out of memory trying to serialize about 50K such records. I'm not sure whether this is a limitation of openamf but it occurs to me that there must be some better way to stream the data over without bloating up memory of my servlet container. Any thoughts on this? Thanks. Ban --- In flexcoders@yahoogroups.com, Abyss Knight [EMAIL PROTECTED] wrote: Have you considered paging the dataset in the ActionScript, but applying the operations to the original ArrayCollection/Dataset? What I mean is, although the grid may only show say 15 records at a time, the dataset is still stored as an AS object and could be adjusted client side, and the current page reloaded. I'm imagining the bottleneck isn't the data structure, but rather the DataGrid rendering time. The other option which has already been mentioned would be to do it server side. You could also roll this into AIR and use the built in SQL Lite implementation for added ease and speed. -- William --- In flexcoders@yahoogroups.com, letterpigeon ban.luc@ wrote: Hi all, Thanks for all your responses. Maybe it'll be better if I described our use case in bit more in details. Our users send us trade file (sometimes has ~50K trades in it currently). And let say all of the trades have an invalid broker code, and all got kicked out, the user will have to go in and change the broker code 1 by 1 that could take them hours. Therefore, the ability for our users to perform mass actions on the dataset (either client side or server side) would be a huge win for us. Once again, thanks all. Ban --- In flexcoders@yahoogroups.com, hank williams hank777@ wrote: On Nov 12, 2007 4:35 AM, Tom Chiverton tom.chiverton@ wrote: On Friday 09 Nov 2007, letterpigeon wrote: Hi all, requirement is for the grid to be able to support up to 100K+ or even 1 million rows Why ? It's utterly impossible for a user to cope with this much data at once, so what's the point ? In general, the question of where to process data is totally legit. If you read carefully, he did not say he wanted to *show* 100k rows at once. He said he wanted to be able to do mass operations on it on the client side. Without understanding his application and architecture I cannot opine on whether it is better to do this on the client or server. But the idea, in applications that may need to scale, of doing more work on the client, is a good one. If you have lots of clients you can indeed save money on server costs by having the clients help with the work. Hank
[flexcoders] Re: Flex grid to serve 100K-1mil rows
Hi all, Thanks for all your responses. Maybe it'll be better if I described our use case in bit more in details. Our users send us trade file (sometimes has ~50K trades in it currently). And let say all of the trades have an invalid broker code, and all got kicked out, the user will have to go in and change the broker code 1 by 1 that could take them hours. Therefore, the ability for our users to perform mass actions on the dataset (either client side or server side) would be a huge win for us. Once again, thanks all. Ban --- In flexcoders@yahoogroups.com, hank williams [EMAIL PROTECTED] wrote: On Nov 12, 2007 4:35 AM, Tom Chiverton [EMAIL PROTECTED] wrote: On Friday 09 Nov 2007, letterpigeon wrote: Hi all, requirement is for the grid to be able to support up to 100K+ or even 1 million rows Why ? It's utterly impossible for a user to cope with this much data at once, so what's the point ? In general, the question of where to process data is totally legit. If you read carefully, he did not say he wanted to *show* 100k rows at once. He said he wanted to be able to do mass operations on it on the client side. Without understanding his application and architecture I cannot opine on whether it is better to do this on the client or server. But the idea, in applications that may need to scale, of doing more work on the client, is a good one. If you have lots of clients you can indeed save money on server costs by having the clients help with the work. Hank
[flexcoders] Re: Flex grid to serve 100K-1mil rows
Have you considered paging the dataset in the ActionScript, but applying the operations to the original ArrayCollection/Dataset? What I mean is, although the grid may only show say 15 records at a time, the dataset is still stored as an AS object and could be adjusted client side, and the current page reloaded. I'm imagining the bottleneck isn't the data structure, but rather the DataGrid rendering time. The other option which has already been mentioned would be to do it server side. You could also roll this into AIR and use the built in SQL Lite implementation for added ease and speed. -- William --- In flexcoders@yahoogroups.com, letterpigeon [EMAIL PROTECTED] wrote: Hi all, Thanks for all your responses. Maybe it'll be better if I described our use case in bit more in details. Our users send us trade file (sometimes has ~50K trades in it currently). And let say all of the trades have an invalid broker code, and all got kicked out, the user will have to go in and change the broker code 1 by 1 that could take them hours. Therefore, the ability for our users to perform mass actions on the dataset (either client side or server side) would be a huge win for us. Once again, thanks all. Ban --- In flexcoders@yahoogroups.com, hank williams hank777@ wrote: On Nov 12, 2007 4:35 AM, Tom Chiverton tom.chiverton@ wrote: On Friday 09 Nov 2007, letterpigeon wrote: Hi all, requirement is for the grid to be able to support up to 100K+ or even 1 million rows Why ? It's utterly impossible for a user to cope with this much data at once, so what's the point ? In general, the question of where to process data is totally legit. If you read carefully, he did not say he wanted to *show* 100k rows at once. He said he wanted to be able to do mass operations on it on the client side. Without understanding his application and architecture I cannot opine on whether it is better to do this on the client or server. But the idea, in applications that may need to scale, of doing more work on the client, is a good one. If you have lots of clients you can indeed save money on server costs by having the clients help with the work. Hank
[flexcoders] Re: Flex grid to serve 100K-1mil rows
Personally i think it very bad idea, can you imagine sort/search 1M records at client, how fast it will be ? What if you decided to do full text search by pattern? your client will die! Can you image how much memory it will consume at client side? best way to do is - smart server querying, pagination and dynamic load. --- In flexcoders@yahoogroups.com, letterpigeon [EMAIL PROTECTED] wrote: Hi all, We've built a flex grid that supports mass action on its data right now (column-wise search replace). But it just doesn't scale right now. It can hold up to around 3K+ rows in the grid, but it's taking a good few mins to load that up from the java backend servlet. The requirement is for the grid to be able to support up to 100K+ or even 1 million rows (may be not displaying them all at once, but when the mass action/search is taken, it should be performed on the complete dataset, not just what is being served up on the grid). We're just looking for ideas how this could be achieved, i.e.: if pagination is needed, what is the best way to do? client side pagination v.s. server? Could this be a good use case for Flex Data Service? etc. Any pointers/ideas/experience would be greatly appreicated. Thanks in advance. Regards, Ban
[flexcoders] Re: Flex grid to serve 100K-1mil rows
BTW, according to my personal experience flash player unable to handle more that 10K records in data grid. Just physically unable - out of memory error! --- In flexcoders@yahoogroups.com, letterpigeon [EMAIL PROTECTED] wrote: Hi all, We've built a flex grid that supports mass action on its data right now (column-wise search replace). But it just doesn't scale right now. It can hold up to around 3K+ rows in the grid, but it's taking a good few mins to load that up from the java backend servlet. The requirement is for the grid to be able to support up to 100K+ or even 1 million rows (may be not displaying them all at once, but when the mass action/search is taken, it should be performed on the complete dataset, not just what is being served up on the grid). We're just looking for ideas how this could be achieved, i.e.: if pagination is needed, what is the best way to do? client side pagination v.s. server? Could this be a good use case for Flex Data Service? etc. Any pointers/ideas/experience would be greatly appreicated. Thanks in advance. Regards, Ban
RE: [flexcoders] Re: Flex grid to serve 100K-1mil rows
BTW, according to my personal experience flash player unable to handle
more that 10K records in data grid.
Just physically unable - out of memory error!
I have no problem populating a DataGrid with one million records. The
following program takes 519 MB when I run it on my Windows machine,
which has 2 GB of RAM. With no records it takes 73 MB, so each record
requires about 450 bytes of memory for the 6 String fields.
- Gordon
?xml version=1.0 encoding=utf-8?
mx:Application xmlns:mx=http://www.adobe.com/2006/mxml
http://www.adobe.com/2006/mxml
initialize=initializeHandler(event)
mx:Script
![CDATA[
import Person;
private function initializeHandler(event:Event):void
{
const N:int = 100;
var a:Array = new Array(N);
for (var i:int = 0; i N; i++)
{
var p:Person = new Person();
p.firstName = FirstName + i;
p.lastName = LirstName + i;
p.address = Address + i;
p.city = City + i;
p.state = State + i;
p.zip = Zip + i;
a[i] = p;
}
dg.dataProvider = a;
}
]]
/mx:Script
mx:DataGrid id=dg
mx:columns
mx:DataGridColumn dataField=firstName/
mx:DataGridColumn dataField=lastName/
mx:DataGridColumn dataField=address/
mx:DataGridColumn dataField=city/
mx:DataGridColumn dataField=state/
mx:DataGridColumn dataField=zip/
/mx:columns
/mx:DataGrid
/mx:Application
-
Person.as:
package
{
public class Person
{
public function Person()
{
super();
}
public var firstName:String;
public var lastName:String;
public var address:String;
public var city:String;
public var state:String;
public var zip:String;
}
}
From: flexcoders@yahoogroups.com [mailto:[EMAIL PROTECTED] On
Behalf Of lytvynyuk
Sent: Friday, November 09, 2007 12:23 PM
To: flexcoders@yahoogroups.com
Subject: [flexcoders] Re: Flex grid to serve 100K-1mil rows
BTW, according to my personal experience flash player unable to handle
more that 10K records in data grid. Just physically unable - out of
memory error!
--- In flexcoders@yahoogroups.com mailto:flexcoders%40yahoogroups.com
, letterpigeon [EMAIL PROTECTED] wrote:
Hi all,
We've built a flex grid that supports mass action on its data right
now (column-wise search replace). But it just doesn't scale right
now. It can hold up to around 3K+ rows in the grid, but it's taking a
good few mins to load that up from the java backend servlet. The
requirement is for the grid to be able to support up to 100K+ or even
1 million rows (may be not displaying them all at once, but when the
mass action/search is taken, it should be performed on the complete
dataset, not just what is being served up on the grid).
We're just looking for ideas how this could be achieved, i.e.: if
pagination is needed, what is the best way to do? client side
pagination v.s. server? Could this be a good use case for Flex Data
Service? etc.
Any pointers/ideas/experience would be greatly appreicated. Thanks in
advance.
Regards,
Ban
