Re: [HACKERS] Extending varlena

2008-08-20 Thread Stephen R. van den Berg
Peter Eisentraut wrote: If you replace the third point by maybe partition TOAST tables, replace large object handle by TOAST pointer, and create an API to work on TOAST pointers, how are the two so much different? And why should they be? I can see that there are going to be needs to access

Re: [HACKERS] Extending varlena

2008-08-20 Thread Tom Lane
Peter Eisentraut wrote: If you replace the third point by maybe partition TOAST tables, replace large object handle by TOAST pointer, and create an API to work on TOAST pointers, how are the two so much different? And why should they be? The reason they should be different is that (IMHO

Re: [HACKERS] Extending varlena

2008-08-19 Thread Jeremy Drake
On Mon, 18 Aug 2008, Tom Lane wrote: What would make more sense is to redesign the large-object stuff to be somewhat modern and featureful, and provide stream-access APIs (think lo_read, lo_seek, etc) that allow offsets wider than 32 bits. A few years ago, I was working on such a project for

Re: [HACKERS] Extending varlena

2008-08-19 Thread Magnus Hagander
Josh Berkus wrote: Andrew, I always find these requests puzzling. Is it really useful to store the data for a jpeg, video file or a 10GB tar ball in a database column? Some people find it useful. Because LOs are actually easier to manage in PG than in most other DBMSes, right now

Re: [HACKERS] Extending varlena

2008-08-19 Thread Magnus Hagander
Simon Riggs wrote: On Mon, 2008-08-18 at 16:22 -0400, Tom Lane wrote: David Fetter [EMAIL PROTECTED] writes: What would need to happen for the next jump up from where varlena is now, to 8 bytes? Dealing with upwards-of-4GB blobs as single Datums isn't remotely sane, and won't become so in

Re: [HACKERS] Extending varlena

2008-08-19 Thread Peter Eisentraut
Am Monday, 18. August 2008 schrieb Tom Lane: - permissions features (more than none anyway) - better management of orphaned objects (obsoleting vacuumlo) - support 16TB of large objects (maybe partition pg_largeobject?) - dump and restore probably need improvement to be practical for such  

Re: [HACKERS] Extending varlena

2008-08-19 Thread Kevin Grittner
Tom Lane [EMAIL PROTECTED] wrote: I was kinda wondering about something closer to the TOAST model, where a blob is only referenceable from a value that's in a table field; and that value encapsulates the name of the blob in some way that needn't even be user-visible. This'd greatly

[HACKERS] Extending varlena

2008-08-18 Thread David Fetter
Folks, As the things stored in databases grow, we're going to start needing to think about database objects that 4 bytes of size can't describe. People are already storing video in lo and bytea fields. To date, the sizes of media files have never trended downward. What would need to happen for

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
David Fetter [EMAIL PROTECTED] writes: What would need to happen for the next jump up from where varlena is now, to 8 bytes? Dealing with upwards-of-4GB blobs as single Datums isn't remotely sane, and won't become so in the near (or even medium) future. So I don't see the point of doing all

Re: [HACKERS] Extending varlena

2008-08-18 Thread David Fetter
On Mon, Aug 18, 2008 at 04:22:56PM -0400, Tom Lane wrote: David Fetter [EMAIL PROTECTED] writes: What would need to happen for the next jump up from where varlena is now, to 8 bytes? Dealing with upwards-of-4GB blobs as single Datums isn't remotely sane, and won't become so in the near

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
David Fetter wrote: Folks, As the things stored in databases grow, we're going to start needing to think about database objects that 4 bytes of size can't describe. People are already storing video in lo and bytea fields. To date, the sizes of media files have never trended downward. I

Re: [HACKERS] Extending varlena

2008-08-18 Thread David Fetter
On Mon, Aug 18, 2008 at 06:09:13PM -0400, Andrew Chernow wrote: David Fetter wrote: Folks, As the things stored in databases grow, we're going to start needing to think about database objects that 4 bytes of size can't describe. People are already storing video in lo and bytea fields. To

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
David Fetter wrote: On Mon, Aug 18, 2008 at 06:09:13PM -0400, Andrew Chernow wrote: David Fetter wrote: Folks, As the things stored in databases grow, we're going to start needing to think about database objects that 4 bytes of size can't describe. People are already storing video in lo and

Re: [HACKERS] Extending varlena

2008-08-18 Thread Josh Berkus
Andrew, I always find these requests puzzling. Is it really useful to store the data for a jpeg, video file or a 10GB tar ball in a database column? Some people find it useful. Because LOs are actually easier to manage in PG than in most other DBMSes, right now that's a significant source

Re: [HACKERS] Extending varlena

2008-08-18 Thread Jeff Davis
I always find these requests puzzling. Is it really useful to store the data for a jpeg, video file or a 10GB tar ball in a database column? One use case is that it can use the existing postgresql protocol, and does not require extra filesystem mounts, extra error handling, and other

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
Andrew Chernow [EMAIL PROTECTED] writes: Anyways (back on topic), I am in favor of removing limits from any section of the database ... not just your suggestion. The end-user application should impose limits. That's nice as an abstract principle, but there are only so many hours in the day,

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
Tom Lane wrote: Andrew Chernow [EMAIL PROTECTED] writes: Anyways (back on topic), I am in favor of removing limits from any section of the database ... not just your suggestion. The end-user application should impose limits. That's nice as an abstract principle, but there are only so many

Re: [HACKERS] Extending varlena

2008-08-18 Thread Andrew Chernow
Jeff Davis wrote: I always find these requests puzzling. Is it really useful to store the data for a jpeg, video file or a 10GB tar ball in a database column? One use case is that it can use the existing postgresql protocol, So can what I am suggesting. How about a user-defined C

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
David Fetter [EMAIL PROTECTED] writes: On Mon, Aug 18, 2008 at 04:22:56PM -0400, Tom Lane wrote: The main things I think we'd need to consider besides just the access API are - permissions features (more than none anyway) Would ROLEs work, or are you thinking of the per-row and per-column

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
David Fetter [EMAIL PROTECTED] writes: On Mon, Aug 18, 2008 at 06:09:13PM -0400, Andrew Chernow wrote: I always find these requests puzzling. Is it really useful to store the data for a jpeg, video file or a 10GB tar ball in a database column? It is if you need transaction semantics.

Re: [HACKERS] Extending varlena

2008-08-18 Thread David Fetter
On Mon, Aug 18, 2008 at 07:31:04PM -0400, Tom Lane wrote: David Fetter [EMAIL PROTECTED] writes: On Mon, Aug 18, 2008 at 04:22:56PM -0400, Tom Lane wrote: The main things I think we'd need to consider besides just the access API are - permissions features (more than none anyway)

Re: [HACKERS] Extending varlena

2008-08-18 Thread Gregory Williamson
David Fetter wrote ... This'd greatly simplify the cleanup-dead-objects problem, and we could avoid addressing the permissions problem at all, since regular SQL permissions on the table would serve fine. But it's not clear what regular SQL fetch and update behaviors should be like

Re: [HACKERS] Extending varlena

2008-08-18 Thread Simon Riggs
On Mon, 2008-08-18 at 16:22 -0400, Tom Lane wrote: David Fetter [EMAIL PROTECTED] writes: What would need to happen for the next jump up from where varlena is now, to 8 bytes? Dealing with upwards-of-4GB blobs as single Datums isn't remotely sane, and won't become so in the near (or even

Re: [HACKERS] Extending varlena

2008-08-18 Thread Tom Lane
Simon Riggs [EMAIL PROTECTED] writes: Probably also using a separate Sequence to allocate numbers rather than using up all the Oids on LOs would be a good plan. Well, assuming that your Large Objects are actually Large, you aren't going to need as many OIDs as all that ;-) However: I was

Re: [HACKERS] Extending varlena

2008-08-18 Thread Simon Riggs
On Mon, 2008-08-18 at 23:43 -0400, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: Probably also using a separate Sequence to allocate numbers rather than using up all the Oids on LOs would be a good plan. Well, assuming that your Large Objects are actually Large, you aren't going