Re: db-create with PostgreSQL fails with user "user"

[Mads Kiilerich]

On 5/15/20 9:35 PM, Louis Bertrand wrote:
I chose PostgreSQL because a) it's already installed on the eventual 
target server and b) I'm familiar with it. However, where is the 
trade-off between SQLite and PostgreSQL? Tens, hundreds, thousands? 
Number of users, transactions? Etc. In other words, I might have saved 
myself some trouble by simply accepting the default for a small site. 



One advantage of supporting multiple databases is that we don't have to 
choose on behalf of someone else ;-)


Also, I guess the load of a "user" might vary with a factor 100 
depending on how the system is used. I guess a system with 1000 quite 
active users easily can have less than 1 write operation per second and 
a manageable amount of read operations and work fine with a number of 
worker processes on a single server. In that case I guess sqlite would 
work fine.


I think the biggest difference is in the operations. If a DBA is running 
the system, he might want a "real" database. There might be more 
confidence in how PostgreSQL handle scaling and system failures and 
allow more powerful integrations with other systems. It is very hard to 
lose critical data in a DVCS hosting system. The important data are in 
the repos directly in the file system - not in the database. And worst 
case, all the clients will still have the data they pushed ... and other 
clients might already have pulled them.


We could perhaps put some generic advice in the docs:

Start out with a single server with sqlite and reliable local storage 
(with regular backup of storage and database). If you need failover, 
make it cold. If users experience delays while the server load is low 
(especially if the workers are busy serving repo data), add more worker 
processes. If the server load is high, consider adding more CPU or 
memory if feasible. If network load is the bottleneck, try solving that. 
Also consider offloading buildfarm load to a simpler or separate system 
or make it "smarter" to decrease the load.


Then, if necessary or for other reasons desirable, scale up to use 
something like PostgreSQL (possibly on a separate HA system/cluster ... 
and potentially higher latency), shared storage (network or SAN), and 
multiple (physical) worker servers.


While different existing setups might have met different limitations, 
does any existing users have any comments or advice to add to this?


/Mads

___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

RE: db-create with PostgreSQL fails with user "user"

[Louis Bertrand]

-Original Message-
From: Mads Kiilerich 
Sent: May 15, 2020 12:56 PM
To: Louis Bertrand ; Thomas De Schampheleire 

Cc: Kallithea 
Subject: Re: db-create with PostgreSQL fails with user "user"

[EXTERNAL EMAIL]

On 5/15/20 10:08 AM, Louis Bertrand wrote:
> Hi Thomas,
>
> Thanks for the prompt reply but it turns out that it's pilot error.
> The name 'user' came from my.ini, where the postgresql URL was set to
> default. After setting the URL correctly
>
> sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea
>(not the real password)
>
> the database seems to be initialized correctly.


Would a more verbose installation overview have helped you? Something like
https://kallithea-scm.org/repos/kallithea-incoming/changeset/512aef4199733151ec76490915bca1e6ce7d63e2
?


/Mads



Hi Mads,
Yes, greatly (except for slight typo s/scheme/schema). I chose PostgreSQL 
because a) it's already installed on the eventual target server and b) I'm 
familiar with it. However, where is the trade-off between SQLite and 
PostgreSQL? Tens, hundreds, thousands? Number of users, transactions? Etc. In 
other words, I might have saved myself some trouble by simply accepting the 
default for a small site.
Thanks
 --Louis





This message is intended only for the named recipients. This message may 
contain information that is confidential or exempt from disclosure under 
applicable law. Any dissemination or copying of this message by anyone other 
than a named recipient is strictly prohibited. If you are not a named recipient 
or an employee or agent responsible for delivering this message to a named 
recipient, please notify us immediately, and permanently destroy this message 
and any copies you may have. Warning: Email may not be secure unless properly 
encrypted.
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: db-create with PostgreSQL fails with user "user"

[Mads Kiilerich]

On 5/15/20 10:08 AM, Louis Bertrand wrote:

Hi Thomas,

Thanks for the prompt reply but it turns out that it's pilot error. The name 
'user' came from my.ini, where the postgresql URL was set to default. After 
setting the URL correctly

sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea
   (not the real password)

the database seems to be initialized correctly.



Would a more verbose installation overview have helped you? Something 
like 
https://kallithea-scm.org/repos/kallithea-incoming/changeset/512aef4199733151ec76490915bca1e6ce7d63e2 
?



/Mads


___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general

Re: db-create with PostgreSQL fails with user "user"

[Louis Bertrand]

Hi Thomas,
I'm not familiar with how Raspberry Pi or Debian package things, but at first 
glance the two packages do indeed seem to be coming from different 
distributions.
Thanks anyway
 --Louis


From: Thomas De Schampheleire 
Sent: Friday, May 15, 2020 8:45 AM
To: Louis Bertrand
Cc: Kallithea
Subject: Re: db-create with PostgreSQL fails with user "user"

[EXTERNAL EMAIL]

Hi Louis,

Ok, great you found the postgreql error.

Regarding npm: it seems indeed your nodejs is too new for the npm version.
Based on the versions:

nodejs (10.19.0~dfsg1-1+rpi1).
npm (5.8.0+ds6-4+deb10u1).

It seems that the nodejs installation is rpi specific. Is it coming
from the same source as the npm package?
Normally, a distribution like debian will provide a consistent set of
packages that work together, but if packages from several sources are
mixed, then such problem could occur. Or of course it could be a
problem in Raspbian itself.
In any case, this is not something that Kallthea decides or interferes
with, so I'm afraid there is not much we can do.

Hope this helps,
Thomas



El vie., 15 may. 2020 a las 10:08, Louis Bertrand
() escribió:
>
> Hi Thomas,
>
> Thanks for the prompt reply but it turns out that it's pilot error. The name 
> 'user' came from my.ini, where the postgresql URL was set to default. After 
> setting the URL correctly
>
>sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea
>   (not the real password)
>
> the database seems to be initialized correctly.
>
>
> However the next step crashes on 'npm install' (see text below).
>
> With apt-get I can confirm that npm and nodejs are at the latest available 
> versions but that seems to be the problem:
>
>   nodejs is already the newest version (10.19.0~dfsg1-1+rpi1).
>
>   npm is already the newest version (5.8.0+ds6-4+deb10u1).
> npm does not recognize version NodeJS 10.x as valid.
>
>
> I should mention here that I am using a Raspberry Pi mostly because it's what 
> I have on hand. I'm trying to prototype the system so that I can get our 
> college IT dept to install it on a public facing server for student work. The 
> college server is running Ubuntu 18.04.4 LTS. The Pi is running Raspbian 10 
> (buster). If the Pi is not suitable or supported, I can switch to a VM 
> running internally (access via VPN).
>
>
> Thanks in advance
>
>  --Louis
>
>
>
> Full text of output:
>
> (kallithea-venv) kallithea@raspberrypi:~/kallithea$ kallithea-cli 
> front-end-build
> Running 'npm install' to install front-end dependencies from package.json
> npm WARN npm npm does not support Node.js v10.19.0
> npm WARN npm You should probably upgrade to a newer version of node as we
> npm WARN npm can't make any promises that npm will work with this version.
> npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 
> 8, 9.
> npm WARN npm You can find the latest version at https://nodejs.org/
> npm WARN notice [SECURITY] bootstrap has the following vulnerability: 1 
> moderate. Go here for more details: 
> https://www.npmjs.com/advisories?search=bootstrap=3.3.7 - Run `npm i 
> npm@latest -g` to upgrade your npm version, and then `npm audit` to get more 
> info.
> npm WARN tar write after end
> npm WARN notice [SECURITY] clean-css has the following vulnerability: 1 low. 
> Go here for more details: 
> https://www.npmjs.com/advisories?search=clean-css=3.4.28 - Run `npm i 
> npm@latest -g` to upgrade your npm version, and then `npm audit` to get more 
> info.
> npm WARN tar write after end
> npm WARN tar write after end
> npm ERR! cb() never called!
>
> npm ERR! This is an error with npm itself. Please report this error at:
> npm ERR! 
>
> npm ERR! A complete log of this run can be found in:
> npm ERR! /var/lib/kallithea/.npm/_logs/2020-05-15T07_32_43_524Z-debug.log
> Traceback (most recent call last):
>   File "/var/lib/kallithea/kallithea-venv/bin/kallithea-cli", line 11, in 
> 
> load_entry_point('Kallithea', 'console_scripts', 'kallithea-cli')()
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 829, in __call__
> return self.main(*args, **kwargs)
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 782, in main
> rv = self.invoke(ctx)
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 1259, in invoke
> return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 1066, in invoke
> return ctx.invoke(self.callback, **ctx.params)
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 610, in invoke
> return callback(*args, **kwargs)
>   File 
> "/var/lib/kallithea/kallithea/kallithea/bin/kallithea_cli_front_end.py", line 
> 46, in front_end_build
> 

Re: db-create with PostgreSQL fails with user "user"

[Thomas De Schampheleire]

Hi Louis,

Ok, great you found the postgreql error.

Regarding npm: it seems indeed your nodejs is too new for the npm version.
Based on the versions:

nodejs (10.19.0~dfsg1-1+rpi1).
npm (5.8.0+ds6-4+deb10u1).

It seems that the nodejs installation is rpi specific. Is it coming
from the same source as the npm package?
Normally, a distribution like debian will provide a consistent set of
packages that work together, but if packages from several sources are
mixed, then such problem could occur. Or of course it could be a
problem in Raspbian itself.
In any case, this is not something that Kallthea decides or interferes
with, so I'm afraid there is not much we can do.

Hope this helps,
Thomas



El vie., 15 may. 2020 a las 10:08, Louis Bertrand
() escribió:
>
> Hi Thomas,
>
> Thanks for the prompt reply but it turns out that it's pilot error. The name 
> 'user' came from my.ini, where the postgresql URL was set to default. After 
> setting the URL correctly
>
>sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea
>   (not the real password)
>
> the database seems to be initialized correctly.
>
>
> However the next step crashes on 'npm install' (see text below).
>
> With apt-get I can confirm that npm and nodejs are at the latest available 
> versions but that seems to be the problem:
>
>   nodejs is already the newest version (10.19.0~dfsg1-1+rpi1).
>
>   npm is already the newest version (5.8.0+ds6-4+deb10u1).
> npm does not recognize version NodeJS 10.x as valid.
>
>
> I should mention here that I am using a Raspberry Pi mostly because it's what 
> I have on hand. I'm trying to prototype the system so that I can get our 
> college IT dept to install it on a public facing server for student work. The 
> college server is running Ubuntu 18.04.4 LTS. The Pi is running Raspbian 10 
> (buster). If the Pi is not suitable or supported, I can switch to a VM 
> running internally (access via VPN).
>
>
> Thanks in advance
>
>  --Louis
>
>
>
> Full text of output:
>
> (kallithea-venv) kallithea@raspberrypi:~/kallithea$ kallithea-cli 
> front-end-build
> Running 'npm install' to install front-end dependencies from package.json
> npm WARN npm npm does not support Node.js v10.19.0
> npm WARN npm You should probably upgrade to a newer version of node as we
> npm WARN npm can't make any promises that npm will work with this version.
> npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 
> 8, 9.
> npm WARN npm You can find the latest version at https://nodejs.org/
> npm WARN notice [SECURITY] bootstrap has the following vulnerability: 1 
> moderate. Go here for more details: 
> https://www.npmjs.com/advisories?search=bootstrap=3.3.7 - Run `npm i 
> npm@latest -g` to upgrade your npm version, and then `npm audit` to get more 
> info.
> npm WARN tar write after end
> npm WARN notice [SECURITY] clean-css has the following vulnerability: 1 low. 
> Go here for more details: 
> https://www.npmjs.com/advisories?search=clean-css=3.4.28 - Run `npm i 
> npm@latest -g` to upgrade your npm version, and then `npm audit` to get more 
> info.
> npm WARN tar write after end
> npm WARN tar write after end
> npm ERR! cb() never called!
>
> npm ERR! This is an error with npm itself. Please report this error at:
> npm ERR! 
>
> npm ERR! A complete log of this run can be found in:
> npm ERR! /var/lib/kallithea/.npm/_logs/2020-05-15T07_32_43_524Z-debug.log
> Traceback (most recent call last):
>   File "/var/lib/kallithea/kallithea-venv/bin/kallithea-cli", line 11, in 
> 
> load_entry_point('Kallithea', 'console_scripts', 'kallithea-cli')()
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 829, in __call__
> return self.main(*args, **kwargs)
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 782, in main
> rv = self.invoke(ctx)
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 1259, in invoke
> return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 1066, in invoke
> return ctx.invoke(self.callback, **ctx.params)
>   File 
> "/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py",
>  line 610, in invoke
> return callback(*args, **kwargs)
>   File 
> "/var/lib/kallithea/kallithea/kallithea/bin/kallithea_cli_front_end.py", line 
> 46, in front_end_build
> subprocess.check_call(['npm', 'install'], cwd=front_end_dir, 
> shell=kallithea.is_windows)
>   File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
> raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['npm', 'install']' returned non-zero 
> exit status 1
>
>
>
> 
> From: Thomas De Schampheleire 
> Sent: Friday, May 15, 2020 2:03 AM
> To: Louis 

Re: db-create with PostgreSQL fails with user "user"

[Louis Bertrand]

Hi Thomas,

Thanks for the prompt reply but it turns out that it's pilot error. The name 
'user' came from my.ini, where the postgresql URL was set to default. After 
setting the URL correctly

   sqlalchemy.url = postgresql://kallithea:S3CR3T@localhost/kallithea
  (not the real password)

the database seems to be initialized correctly.


However the next step crashes on 'npm install' (see text below).

With apt-get I can confirm that npm and nodejs are at the latest available 
versions but that seems to be the problem:

  nodejs is already the newest version (10.19.0~dfsg1-1+rpi1).

  npm is already the newest version (5.8.0+ds6-4+deb10u1).
npm does not recognize version NodeJS 10.x as valid.


I should mention here that I am using a Raspberry Pi mostly because it's what I 
have on hand. I'm trying to prototype the system so that I can get our college 
IT dept to install it on a public facing server for student work. The college 
server is running Ubuntu 18.04.4 LTS. The Pi is running Raspbian 10 (buster). 
If the Pi is not suitable or supported, I can switch to a VM running internally 
(access via VPN).


Thanks in advance

 --Louis



Full text of output:

(kallithea-venv) kallithea@raspberrypi:~/kallithea$ kallithea-cli 
front-end-build
Running 'npm install' to install front-end dependencies from package.json
npm WARN npm npm does not support Node.js v10.19.0
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 
8, 9.
npm WARN npm You can find the latest version at https://nodejs.org/
npm WARN notice [SECURITY] bootstrap has the following vulnerability: 1 
moderate. Go here for more details: 
https://www.npmjs.com/advisories?search=bootstrap=3.3.7 - Run `npm i 
npm@latest -g` to upgrade your npm version, and then `npm audit` to get more 
info.
npm WARN tar write after end
npm WARN notice [SECURITY] clean-css has the following vulnerability: 1 low. Go 
here for more details: 
https://www.npmjs.com/advisories?search=clean-css=3.4.28 - Run `npm i 
npm@latest -g` to upgrade your npm version, and then `npm audit` to get more 
info.
npm WARN tar write after end
npm WARN tar write after end
npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR! 

npm ERR! A complete log of this run can be found in:
npm ERR! /var/lib/kallithea/.npm/_logs/2020-05-15T07_32_43_524Z-debug.log
Traceback (most recent call last):
  File "/var/lib/kallithea/kallithea-venv/bin/kallithea-cli", line 11, in 

load_entry_point('Kallithea', 'console_scripts', 'kallithea-cli')()
  File 
"/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", 
line 829, in __call__
return self.main(*args, **kwargs)
  File 
"/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", 
line 782, in main
rv = self.invoke(ctx)
  File 
"/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", 
line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
  File 
"/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", 
line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
  File 
"/var/lib/kallithea/kallithea-venv/lib/python3.7/site-packages/click/core.py", 
line 610, in invoke
return callback(*args, **kwargs)
  File "/var/lib/kallithea/kallithea/kallithea/bin/kallithea_cli_front_end.py", 
line 46, in front_end_build
subprocess.check_call(['npm', 'install'], cwd=front_end_dir, 
shell=kallithea.is_windows)
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['npm', 'install']' returned non-zero 
exit status 1




From: Thomas De Schampheleire 
Sent: Friday, May 15, 2020 2:03 AM
To: Louis Bertrand
Cc: Kallithea
Subject: Re: db-create with PostgreSQL fails with user "user"

[EXTERNAL EMAIL]
Hello,

Could you provide the full stacktrace please?

Thanks,
Thomas

On Fri, May 15, 2020, 04:14 Louis Bertrand 
mailto:louis.bertr...@durhamcollege.ca>> wrote:
Hello,
I get a strange exception when trying to create the database for a fresh 
installation of Kallithea. The exception complains about unknown user "user" 
even though I specified --user=Kallithea on the command line.

The full command line, run as user Kallithea:
$ kallithea-cli db-create -c my.ini --user=kallithea --password=blahblah 
--email=kallit...@domain.tld --repos=/var/lib/kallithea/repos

The exception seems to be raised by psycopg2 when connecting to the database, 
and called from sqlalchemy:
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: FATAL:  password authentication failed for user 
"user"

The user kallithea exists as 

Re: db-create with PostgreSQL fails with user "user"

[Thomas De Schampheleire]

Hello,

Could you provide the full stacktrace please?

Thanks,
Thomas

On Fri, May 15, 2020, 04:14 Louis Bertrand 
wrote:

> Hello,
> I get a strange exception when trying to create the database for a fresh
> installation of Kallithea. The exception complains about unknown user
> "user" even though I specified --user=Kallithea on the command line.
>
> The full command line, run as user Kallithea:
> $ kallithea-cli db-create -c my.ini --user=kallithea --password=blahblah
> --email=kallit...@domain.tld --repos=/var/lib/kallithea/repos
>
> The exception seems to be raised by psycopg2 when connecting to the
> database, and called from sqlalchemy:
> conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
> psycopg2.OperationalError: FATAL:  password authentication failed for user
> "user"
>
> The user kallithea exists as a PostgreSQL role and has createdb
> permission, but it seems that the user name doesn't get through to
> sqlalchemy because it's using "user" instead of the name supplied on the
> command line.
>
> Any ideas? Full rollback available if you wish.
>
> Thanks
>  --Louis
>
>
> Louis Bertrand, P.Eng.
> Professor, School of Science and Engineering Technology
> Durham College, Oshawa ON Canada
>
> 
>
> 
> This message is intended only for the named recipients. This message may
> contain information that is confidential or exempt from disclosure under
> applicable law. Any dissemination or copying of this message by anyone
> other than a named recipient is strictly prohibited. If you are not a named
> recipient or an employee or agent responsible for delivering this message
> to a named recipient, please notify us immediately, and permanently destroy
> this message and any copies you may have. Warning: Email may not be secure
> unless properly encrypted.
> ___
> kallithea-general mailing list
> kallithea-general@sfconservancy.org
> https://lists.sfconservancy.org/mailman/listinfo/kallithea-general
>
___
kallithea-general mailing list
kallithea-general@sfconservancy.org
https://lists.sfconservancy.org/mailman/listinfo/kallithea-general