subject:"RES\: Is there any other way\? DataProviders must hit the Db twice for \(possible\) large datasets"

RES: Is there any other way? DataProviders must hit the Db twice for (possible) large datasets

2008-11-26 Thread Bruno Cesar Borges

You have to consider some things

1) Is this table going to be huge?
2) Am I going to let users do a Full Table Scan? (letting them search without
filters, for instance)
3) Is this table frequently updated? How often new data is added to it?
4) Any other question that may affect your scalability ...

Then you will know if is a good idea to just bring the whole data, or do a
count(*) before.

But, from my experience I rather say: always do a count(*) before.

Regards,
Bruno

-Mensagem original-
De: Igor Vaynberg [mailto:[EMAIL PROTECTED]
Enviada em: quarta-feira, 26 de novembro de 2008 15:20
Para: users@wicket.apache.org
Assunto: Re: Is there any other way? DataProviders must hit the Db twice
for (possible) large datasets

On Wed, Nov 26, 2008 at 9:06 AM, Wayne Pope
[EMAIL PROTECTED] wrote:
Hi Igor,

what? why would you ever load the whole dataset?
just to avoid 2 calls on smallish datasets, especially when there are
multiple joins and database isnt on the same box.

so you think pushing all that extra data over the network is actually
more efficient then doing another query wtf.

yeah. because select count() queries are the most expensive queries
you can run on the database. you are right, its totally going to kill
it. you know how all those sites on the internet that have a pager
above the pageable view that shows you the number of the last
available page...you know how those work without doing a select
count()?

Ouch.
I just want to limit calls if possible to the database as waiting for i/o's
is never great for scalability. I'm not 'having a go' at wicket or DataViews
or anything, just trying to understand it. I never claimed to be a guru -
far from it.

i can only assume that you have actually profiled your app and that
one select count() call was what was taking a significant chunk of
processing time in the database server? to the point where eliminating
it will actually reduce enough load on the database to increase your
throughput?

-igor

Wayne

On Wed, Nov 26, 2008 at 5:58 PM, Igor Vaynberg [EMAIL PROTECTED]wrote:

On Wed, Nov 26, 2008 at 7:32 AM, Wayne Pope
[EMAIL PROTECTED] wrote:
I'm sure I must be missing something still, as I can't beleive that we
need
to either a) load the whole data set

what? why would you ever load the whole dataset?

b) call count on the Db , then load in
the iterator mehod. Thats going to kill the database in prod (or really
not
help.)

-igor

On Wed, Nov 26, 2008 at 3:58 PM, Michael Sparer [EMAIL PROTECTED]
wrote:

have a look at https://issues.apache.org/jira/browse/WICKET-1784

Wayne Pope-2 wrote:

Ok,

I was just having a bit of code clean up and I realized that in our
IDataProviders we are loading all rows for a given dataset.
So looking at the iterator method I see we can limit the result (and
the
offset). Great I thought - however I see that that the size() method
is
called as part of the getViewSize() in the AbstractPageableView. Thus
I
need
to call the database here to figure out the size.

Am I doing sonething wrong or have I got to hit the database twice for
each
DataProvider render.

Obvously I don't want to hard code a size. Is there any other way ?

Thanks
Wayne

-
Michael Sparer
http://talk-on-tech.blogspot.com
--
View this message in context:

http://www.nabble.com/Is-there-any-other-way--DataProviders-must-hit-the-Db-twice-for-%28possible%29-large-datasets-tp20701684p20702476.html
Sent from the Wicket - User mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

***
Atenção: Esta mensagem foi enviada para uso exclusivo do(s) destinatários(s)
acima identificado(s),
podendo conter informações e/ou documentos confidencias/privilegiados e seu
sigilo é protegido por
lei. Caso você tenha recebido por engano, por favor, informe o remetente e
apague-a de seu sistema.
Notificamos que é proibido por lei a sua retenção, disseminação, distribuição,
cópia ou uso sem
expressa autorização do

RES: Is there any other way? DataProviders must hit the Db twice for (possible) large datasets

2008-11-26 Thread Bruno Cesar Borges

Here's what I've commented on this issue:

I think the purpose of Transfer Object Pattern , where it says it wants to
reduce remote calls between tiers over the network is related to tiers like
client / server architectures.

This means that:

i) A Swing client requests a query on a table for a remote EJB
ii) The EJB executes two calls on the database
ii.a) select count(*)
ii.b) select * ... limit x offset y / rownum between z and k // whatever DB
paging technique is being used.
iii) Put all this information into one single Transfer Object
iv) Send the object back to the client application over the network

So, if your Web Application is running on the same server with your services
(EJBs, Spring Services, whatever), this means that you are not going to
transport data over the network between those tiers (Web and Model). And there
will be an implicit Transfer Object to the real client (web browser).

Following this logic: -1 for this improvement.

Regards,
Bruno

-Mensagem original-
De: rgoodwin [mailto:[EMAIL PROTECTED]
Enviada em: quarta-feira, 26 de novembro de 2008 19:01
Para: users@wicket.apache.org
Assunto: RE: Is there any other way? DataProviders must hit the Db twice
for (possible) large datasets

Still is being disputed unfortunately :(

https://issues.apache.org/jira/browse/WICKET-1784

For those who believe in the 1-query approach, ergo the DTO pattern, writing
an alternative to the Wicket data provision using a DTO approach isn't a
terrible hardship ... just annoying to be sprung with on day one of using
Wicket. But pain is experienced later when encountering the Wicket data
provision and paging mechanism coupling. The paging mechanism is far less
joy to replace ...

IReallyHateDataProvider

Hoover, William wrote:

I think the idea behind this is that size will be called first. If the
size is zero there is no need to proceed with the call to get the items.
I don't necessarily agree with this approach because a lot of service
calls can capture the data in one call (even down to the database level-
some support getting the size/results in one call), but the last time I
brought this issue up it was disputed.

--
View this message in context:
http://www.nabble.com/Is-there-any-other-way--DataProviders-must-hit-the-Db-twice-for-%28possible%29-large-datasets-tp20701684p20708929.html
Sent from the Wicket - User mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Warning: This message was sent for exclusive use of the addressees above
identified, possibly
containing information and or privileged/confidential documents whose content
is protected by law.
In case you have mistakenly received it, please notify the sender and delete it
from your system.
Be noticed that the law forbids the retention, dissemination, distribution,
copy or use without
express authorization from the sender. Personal opinions of the sender do not
necessarily reflect
CETIP's point of view, which is only divulged by authorized personnel.
***

-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RES: Is there any other way? DataProviders must hit the Db twice for (possible) large datasets

RES: Is there any other way? DataProviders must hit the Db twice for (possible) large datasets

2 matches

Site Navigation

Mail list logo

Footer information