Re: [GENERAL] storing large files in database - performance

On Fri, May 19, 2017 at 2:04 PM, Eric Hill wrote: > I am pleased to report that with Merlin's suggestion of using the > pg-large-object middleware, I have a test case now showing that I can write a > 25MB buffer from Node.js to Postgres in roughly 700 milliseconds. Here is

Re: [GENERAL] storing large files in database - performance

I am pleased to report that with Merlin's suggestion of using the pg-large-object middleware, I have a test case now showing that I can write a 25MB buffer from Node.js to Postgres in roughly 700 milliseconds. Here is the JavaScript code, which is nearly verbatim from the example in the

Re: [GENERAL] storing large files in database - performance

Thanks, Merlin - lots of good information here, and I had not yet stumbled across pg-large-object - I will look into it. Eric -Original Message- From: Merlin Moncure [mailto:mmonc...@gmail.com] Sent: Thursday, May 18, 2017 9:49 AM To: Eric Hill Cc: Thomas Kellerer

Re: [GENERAL] storing large files in database - performance

My apologies: I said I ran "this query" but failed to include the query. It was merely this: SELECT "indexFile"."_id", "indexFile"."contents" FROM "mySchema"."indexFiles" AS "indexFile" WHERE "indexFile"."_id" = '591c609bb56d0849404e4720'; Eric -Original Message- From: Eric Hill

Re: [GENERAL] storing large files in database - performance

On Thu, May 18, 2017 at 7:34 AM, Eric Hill wrote: > I would be thrilled to get 76 MB per second, and it is comforting to know > that we have that as a rough upper bound on performance. I've got work to do > to figure out how to approach that upper bound from Node.js. > > In

Re: [GENERAL] storing large files in database - performance

I would be thrilled to get 76 MB per second, and it is comforting to know that we have that as a rough upper bound on performance. I've got work to do to figure out how to approach that upper bound from Node.js. In the meantime, I've been looking at performance on the read side. For that,

Re: [GENERAL] storing large files in database - performance

On Tue, May 16, 2017 at 9:51 AM, Thomas Kellerer wrote: > John R Pierce schrieb am 16.05.2017 um 16:44: >> On 5/16/2017 7:35 AM, Thomas Kellerer wrote: >>> When my (JDBC based) SQL client and the database server are on the same >>> computer... >> >> node.js is Javascript, not

Re: [GENERAL] storing large files in database - performance

On 2017-05-16 12:25:03 +, Eric Hill wrote: > I searched and found a few discussions of storing large files in the database > in the archives, but none that specifically address performance and how large > of files can realistically be stored in the database. > > > > I have a node.js

Re: [GENERAL] storing large files in database - performance

OK, thanks very much. It seems like my process is somehow flawed. I'll try removing some layers and see if I can figure out what is killing the performance. Eric > > Do these numbers surprise you? Are these files just too large for > storage in PostgreSQL to be practical? Could there be

Re: [GENERAL] storing large files in database - performance

John R Pierce schrieb am 16.05.2017 um 16:44: > On 5/16/2017 7:35 AM, Thomas Kellerer wrote: >> When my (JDBC based) SQL client and the database server are on the same >> computer... > > node.js is Javascript, not java w/ jdbc I know that. I mentioned JDBC so that it's clear that the timings

Re: [GENERAL] storing large files in database - performance

On 05/16/2017 07:44 AM, John R Pierce wrote: On 5/16/2017 7:35 AM, Thomas Kellerer wrote: When my (JDBC based) SQL client and the database server are on the same computer... node.js is Javascript, not java w/ jdbc I think it was more a point of comparison, like my using a Python example.

Re: [GENERAL] storing large files in database - performance

On 5/16/2017 5:25 AM, Eric Hill wrote: I do have the Sequelize ORM and the pg driver in between my code and the database. Can you try a similar test without the ORM, just going straight from node.js to sql ? -- john r pierce, recycling bits in santa cruz -- Sent via pgsql-general

Re: [GENERAL] storing large files in database - performance

On 5/16/2017 7:35 AM, Thomas Kellerer wrote: When my (JDBC based) SQL client and the database server are on the same computer... node.js is Javascript, not java w/ jdbc -- john r pierce, recycling bits in santa cruz -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To

Re: [GENERAL] storing large files in database - performance

On 05/16/2017 05:25 AM, Eric Hill wrote: Hey, I searched and found a few discussions of storing large files in the database in the archives, but none that specifically address performance and how large of files can realistically be stored in the database. I have a node.js application using

Re: [GENERAL] storing large files in database - performance

Eric Hill schrieb am 16.05.2017 um 14:25: > I have a node.js application using PostgreSQL to store uploaded > files. The column in which I am storing the file contents is of type > “bytea” with “Storage” type set to “EXTENDED”. Storing a 12.5 MB file > is taking 10 seconds, and storing a 25MB file

Re: [GENERAL] storing large files in database - performance

Eric Hill wrote: > I am storing the file contents is of type "bytea" with "Storage" type set to > "EXTENDED". Storing a 12.5 MB file is taking 10 seconds That seems really slow indeed. Can you import the same file to the same server with psql's \lo_import command and see how much time

[GENERAL] storing large files in database - performance

Hey, I searched and found a few discussions of storing large files in the database in the archives, but none that specifically address performance and how large of files can realistically be stored in the database. I have a node.js application using PostgreSQL to store uploaded files. The