Re: PostgreSQL as advanced job queuing system

2024-03-25 Thread Justin Clift
On 2024-03-25 23:44, Merlin Moncure wrote: On Mon, Mar 25, 2024 at 4:43 AM Dominique Devienne wrote: Hi. Anything you can share? OSS? Doesn't look like it... If it's not, a more details higher level architecture overview would be nice. let me float that, I would love to project-ize this.

Re: PostgreSQL as advanced job queuing system

2024-03-25 Thread Merlin Moncure
On Mon, Mar 25, 2024 at 4:43 AM Dominique Devienne wrote: > On Sat, Mar 23, 2024 at 3:13 AM Merlin Moncure wrote: > >> On Fri, Mar 22, 2024 at 6:58 AM ushi wrote: >> >>> the idea to implement a job queuing system using PostgreSQL. >>> >> >> I wrote an enterprise scheduler, called pgtask, which

Re: PostgreSQL as advanced job queuing system

2024-03-25 Thread Dominique Devienne
On Sat, Mar 23, 2024 at 3:13 AM Merlin Moncure wrote: > On Fri, Mar 22, 2024 at 6:58 AM ushi wrote: > >> the idea to implement a job queuing system using PostgreSQL. >> > > I wrote an enterprise scheduler, called pgtask, which ochestates a very > large amount of work [...] > Hi. Anything you

Re: PostgreSQL as advanced job queuing system

2024-03-22 Thread Merlin Moncure
On Fri, Mar 22, 2024 at 6:58 AM ushi wrote: > Hello List, > > i am playing with the idea to implement a job queuing system using > PostgreSQL. To meet requirements the system needs to offer some advanced > features compared to "classic" queuing systems: > > - users can create new queues at any

Re: PostgreSQL as advanced job queuing system

2024-03-22 Thread Shaheed Haque
Generally, I'd suggest you think carefully about the nature of the jobs, and draw up a list of must-have properties (performance of course, but also things like whether jobs have to survive planned or unplanned outages, be visible across a WAN, numbers of readers and writers, delivery guarantees,

Re: PostgreSQL as advanced job queuing system

2024-03-22 Thread Thiemo Kellner
Am 22.03.2024 um 14:15 schrieb Fred Habash: We developed a home-grown queue system using Postgres, but its performance was largely hindered by que tables bloating and the need to continuously vacuum them. It did not scale whatsoever. With some workarounds, we ended up designing three sets

Re: PostgreSQL as advanced job queuing system

2024-03-22 Thread Fred Habash
We developed a home-grown queue system using Postgres, but its performance was largely hindered by que tables bloating and the need to continuously vacuum them. It did not scale whatsoever. With some workarounds, we ended up designing three sets of queue tables, switching between them based on

Re: PostgreSQL as advanced job queuing system

2024-03-22 Thread Peter J. Holzer
On 2024-03-22 12:43:40 +0100, ushi wrote: > i am playing with the idea to implement a job queuing system using > PostgreSQL. To meet requirements the system needs to offer some advanced > features compared to "classic" queuing systems: > > - users can create new queues at any time > - queues have

Re: PostgreSQL as advanced job queuing system

2024-03-22 Thread Robert Treat
On Fri, Mar 22, 2024 at 8:05 AM Dominique Devienne wrote: > > On Fri, Mar 22, 2024 at 12:58 PM ushi wrote: >> >> i am playing with the idea to implement a job queuing system using >> PostgreSQL. > > > FYI, two bookmarks I have on this subject, that I plan to revisit eventually: > *

Re: PostgreSQL as advanced job queuing system

2024-03-22 Thread Dominique Devienne
On Fri, Mar 22, 2024 at 12:58 PM ushi wrote: > i am playing with the idea to implement a job queuing system using > PostgreSQL. FYI, two bookmarks I have on this subject, that I plan to revisit eventually: * https://news.ycombinator.com/item?id=20020501 *

PostgreSQL as advanced job queuing system

2024-03-22 Thread ushi
Hello List, i am playing with the idea to implement a job queuing system using PostgreSQL. To meet requirements the system needs to offer some advanced features compared to "classic" queuing systems: - users can create new queues at any time - queues have priorities - priorities of queues can