GSoC Proposal - pgAdmin4 Query Tool Automatic Mode Detection

[Yosry Muhammad]

Dear pgAdmin hackers,

I am a student applying to GSoC for the Query Tool Automatic Mode Detection
project. Here is the proposal I submitted:

https://docs.google.com/document/d/1cQssYfPWLSCKAmFwnN-jEPjlldGQtr5fbPaU788UrwM/edit?usp=sharing

Could you please take a look at it and give me your feedback and any
comments you have? I know it's too late but I still have a chance to do
modifications in the next 15 hours.

Thanks. Regards.

-- 
*Yosry Muhammad Yosry*

Computer Engineering student,
The Faculty of Engineering,
Cairo University (2021).
Class representative of CMP 2021.
https://www.linkedin.com/in/yosrym93/

pgAdmin4 v4.5 candidate builds

[Akshay Joshi]

Hi All,

We have released pgAdmin4 v4.4 on last Thursday, but with the release of
psycopg2 v2.8 we have found issue running pgAdmin4 with latest psycopg2. We
have fixed that issue and will plan to release pgAdmin4 v4.5 on coming
Wednesday.

pgAdmin4 v4.5 candidate builds and source can be found at
https://developer.pgadmin.org/builds/2019-04-08-1/

Fahar, can you please verify it for release on Wednesday.

-- 
*Thanks & Regards*
*Akshay Joshi*

*Sr. Software Architect*
*EnterpriseDB Software India Private Limited*
*Mobile: +91 976-788-8246*

pgAdmin 4 commit: Update version for release

[Akshay Joshi]

Update version for release

Branch
--
master

Details
---
https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=ceb0e39e14ecdbdc1805ffe44986bc0bfa2273a3

Modified Files
--
runtime/Info.plist   | 4 ++--
runtime/pgAdmin4.pro | 2 +-
web/config.py| 4 ++--
3 files changed, 5 insertions(+), 5 deletions(-)

pgAdmin 4 commit: Update message catalogs.

[Akshay Joshi]

Update message catalogs.

Branch
--
master

Details
---
https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=3bed332439b7763a3bcb7a3bb6346c53fd9c74f9

Modified Files
--
web/pgadmin/messages.pot   | 7750 +++-
.../translations/de/LC_MESSAGES/messages.mo|  Bin 148353 -> 148249 bytes
.../translations/de/LC_MESSAGES/messages.po| 8377 -
.../translations/es/LC_MESSAGES/messages.mo|  Bin 153165 -> 153061 bytes
.../translations/es/LC_MESSAGES/messages.po| 8149 -
.../translations/fr/LC_MESSAGES/messages.mo|  Bin 157506 -> 157367 bytes
.../translations/fr/LC_MESSAGES/messages.po| 9376 +---
.../translations/ja/LC_MESSAGES/messages.mo|  Bin 164159 -> 164009 bytes
.../translations/ja/LC_MESSAGES/messages.po| 8519 --
.../translations/ko/LC_MESSAGES/messages.mo|  Bin 150815 -> 150711 bytes
.../translations/ko/LC_MESSAGES/messages.po| 8451 --
.../translations/pl/LC_MESSAGES/messages.mo|  Bin 148490 -> 148386 bytes
.../translations/pl/LC_MESSAGES/messages.po| 8477 --
.../translations/ru/LC_MESSAGES/messages.mo|  Bin 185547 -> 185443 bytes
.../translations/ru/LC_MESSAGES/messages.po| 8489 --
.../translations/zh/LC_MESSAGES/messages.mo|  Bin 127755 -> 127651 bytes
.../translations/zh/LC_MESSAGES/messages.po| 8315 -
17 files changed, 14780 insertions(+), 61123 deletions(-)

pgAdmin 4 commit: Updated release notes

[Akshay Joshi]

Updated release notes

Branch
--
master

Details
---
https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=490999f528c5bb63d9956de4d0e43482aed4b1e5

Modified Files
--
docs/en_US/release_notes_4_5.rst | 5 -
1 file changed, 5 deletions(-)

Re: [pgAdmin4][Patch]: Psycopg2 v2.8 fixes

[Akshay Joshi]

Thanks patch applied.

On Mon, Apr 8, 2019 at 5:07 PM Khushboo Vashi <
khushboo.va...@enterprisedb.com> wrote:

> Hi,
>
> I just get to know that, psycopg2 v2.8.1 released because of the below
> issue which we also encountered.
> https://github.com/psycopg/psycopg2/issues/887
>
> To fix the above-mentioned issue, I have made some changes which are not
> required now.
> So, please find the attached updated patch.
>
> Hope this would be the last fix. :)
>
> NOTE: pgAdmin will work with psycopg2* v2.8.1* but not with the *v2.8.*
>
> Thanks,
> Khushboo
>
>
> On Mon, Apr 8, 2019 at 4:28 PM Khushboo Vashi <
> khushboo.va...@enterprisedb.com> wrote:
>
>> Hi,
>>
>> Please find the attached updated patch. I have disabled the Connection
>> status, so before I didn't receive this issue.
>>
>> Thanks,
>> Khushboo
>>
>> On Mon, Apr 8, 2019 at 11:36 AM Khushboo Vashi <
>> khushboo.va...@enterprisedb.com> wrote:
>>
>>> Hi,
>>>
>>> Please find the attached patch to make pgAdmin compatible with psycopg2
>>> v 2.8.
>>> I have also made sure, pgAdmin will work with the older version of
>>> psycopg2.
>>>
>>>
>> Thanks,
>>> Khushboo
>>>
>>

-- 
*Thanks & Regards*
*Akshay Joshi*

*Sr. Software Architect*
*EnterpriseDB Software India Private Limited*
*Mobile: +91 976-788-8246*

pgAdmin 4 commit: Ensure that pgAdmin4 should work properly with psycop

[Akshay Joshi]

Ensure that pgAdmin4 should work properly with psycopg2 v2.8. Fixes #4143

Branch
--
master

Details
---
https://git.postgresql.org/gitweb?p=pgadmin4.git;a=commitdiff;h=c21ea3c3423142410b7b8f43e0e6a6515516d865
Author: Khushboo Vashi 

Modified Files
--
docs/en_US/release_notes_4_5.rst|  7 +--
requirements.txt|  2 +-
web/pgadmin/utils/driver/psycopg2/cursor.py | 22 +-
3 files changed, 27 insertions(+), 4 deletions(-)

Re: [pgAdmin4][Patch]: Psycopg2 v2.8 fixes

[Khushboo Vashi]

Hi,

I just get to know that, psycopg2 v2.8.1 released because of the below
issue which we also encountered.
https://github.com/psycopg/psycopg2/issues/887

To fix the above-mentioned issue, I have made some changes which are not
required now.
So, please find the attached updated patch.

Hope this would be the last fix. :)

NOTE: pgAdmin will work with psycopg2* v2.8.1* but not with the *v2.8.*

Thanks,
Khushboo


On Mon, Apr 8, 2019 at 4:28 PM Khushboo Vashi <
khushboo.va...@enterprisedb.com> wrote:

> Hi,
>
> Please find the attached updated patch. I have disabled the Connection
> status, so before I didn't receive this issue.
>
> Thanks,
> Khushboo
>
> On Mon, Apr 8, 2019 at 11:36 AM Khushboo Vashi <
> khushboo.va...@enterprisedb.com> wrote:
>
>> Hi,
>>
>> Please find the attached patch to make pgAdmin compatible with psycopg2 v
>> 2.8.
>> I have also made sure, pgAdmin will work with the older version of
>> psycopg2.
>>
>>
> Thanks,
>> Khushboo
>>
>


psycopg2_2.8_fixes_v2.patch
Description: Binary data

Re: [pgAdmin4][Patch]: Psycopg2 v2.8 fixes

[Khushboo Vashi]

Hi,

Please find the attached updated patch. I have disabled the Connection
status, so before I didn't receive this issue.

Thanks,
Khushboo

On Mon, Apr 8, 2019 at 11:36 AM Khushboo Vashi <
khushboo.va...@enterprisedb.com> wrote:

> Hi,
>
> Please find the attached patch to make pgAdmin compatible with psycopg2 v
> 2.8.
> I have also made sure, pgAdmin will work with the older version of
> psycopg2.
>
>
Thanks,
> Khushboo
>


psycopg2_2.8_fixes_v1.patch
Description: Binary data

Re: GSoC Student - Help needed

[Dave Page]

On Fri, Apr 5, 2019 at 8:22 PM Yosry Muhammad  wrote:

> Dear pgadmin developers,
>
> I am a student applying to GSoC this year, specifically the query tool
> automatic mode switching project.
>
> I just finished my exams so I am short in time. I would very much
> appreciate it if anyone is willing to contact me privately to give me an
> overview of how the sql editor works and help me understand the relevant
> parts of the code if I get stuck. I already cloned the project and ran it
> successfully and I am a user of pgadmin.
>

I would strongly suggest asking any specific questions you have here on the
mailing list. Any work or discussion should be done publicly so others can
benefit from it.

You are, as you note, very short on time though. The application deadline
is tomorrow, and I know it's unlikely I'll have time to answer much between
now and then as I already have a full schedule for today and most of
tomorrow. I can't speak for others though.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: [pgAdmin4][Patch] - RM 3605 & 2392 - View data fixes

[Khushboo Vashi]

Hi,

Please find the attached updated patch.

Thanks,
Khushboo

On Fri, Apr 5, 2019 at 11:32 AM Akshay Joshi 
wrote:

> Hi Khushboo
>
> #3605 is fixed and found below issue in #2392
>
>- As per steps given in RM, I have perform the same. But when I delete
>the row, delete button is still enabled and save button is disabled even
>though newly created row is still there. Apart from that code looks good to
>me. Please fix it.
>
>
> Fixed.

> On Thu, Apr 4, 2019 at 5:27 PM Khushboo Vashi <
> khushboo.va...@enterprisedb.com> wrote:
>
>> Hi,
>>
>> Please find the attached patch to fix:
>>
>> 1. #3605 - Edit Data: Deleting N number of rows makes first N number of
>> rows disable
>> 2. #2392 - Datagrid: If we delete any row it also saves newly added row
>>
>> Thanks,
>> Khushboo
>>
>>
>>
>
> --
> *Thanks & Regards*
> *Akshay Joshi*
>
> *Sr. Software Architect*
> *EnterpriseDB Software India Private Limited*
> *Mobile: +91 976-788-8246*
>


RM_3605_2392_v1.patch
Description: Binary data

Re: Async queries pretending to be synchronous

[Dave Page]

Done.

On Mon, Apr 8, 2019 at 9:27 AM Khushboo Vashi <
khushboo.va...@enterprisedb.com> wrote:

>
>
> On Tue, Mar 5, 2019 at 10:13 PM Dave Page  wrote:
>
>>
>>
>> On Tue, Mar 5, 2019 at 6:36 AM Khushboo Vashi <
>> khushboo.va...@enterprisedb.com> wrote:
>>
>>>
>>>
>>> On Mon, Mar 4, 2019 at 4:05 PM Khushboo Vashi <
>>> khushboo.va...@enterprisedb.com> wrote:
>>>


 On Mon, Mar 4, 2019 at 4:00 PM Dave Page  wrote:

>
>
> On Mon, Mar 4, 2019 at 10:18 AM Khushboo Vashi <
> khushboo.va...@enterprisedb.com> wrote:
>
>>
>>
>> On Mon, Mar 4, 2019 at 3:43 PM Dave Page  wrote:
>>
>>> Hi
>>>
>>> On Mon, Mar 4, 2019 at 5:43 AM Khushboo Vashi <
>>> khushboo.va...@enterprisedb.com> wrote:
>>>


 On Fri, Mar 1, 2019 at 10:01 PM Dave Page 
 wrote:

> In investigating #3656 I found the initial problem to be that when
> running in a container, Gunicorn will kill the worker process if a 
> thread
> doesn't respond for 30 seconds by default. I fixed that by making the
> timeout match the application session timeout, but it revealed another
> issue.
>
> Given the function below (from the ticket), if you open the query
> tool and run:
>
> SELECT 1; SELECT fails_after(30);
>
> the async query actually blocks for 30 seconds in the
> cur.execute() call in execute_async() in connection.py (line 968). 
> This
> causes the entire app to hang (watch the dashboard requests pile up in
> pending state in the network tab of the browser dev tools).
>
> If you run just the second SELECT, it works as expected, as does
> running something like:
>
> SELECT 1; SELECT pg_sleep(30);
>
> Anyone have any idea what's going on?
>

 Connection.poll() blocking the call here. ( connection.py -  
 _wait_timeout
 function line #1378 state = conn.poll() )
 In the asynchronous connection, after executing the query, the
 conn.poll() is being called to fetch the connection status.
 It gives the status accordingly but in this case, it is blocking
 and not giving the status.

>>>
>>> If I put a breakpoint on the _wait_timeout call (line 969 in
>>> execute_async), it hits it *after* the 30 seconds has passed (during 
>>> which
>>> time all other queries for cancel or dashboards and status etc don't get
>>> processed).
>>>
>>> Put the breakpoint  in the _wait_timeout function itself. The line
>> no 1378  (state = conn.poll()) is blocking the execution 30 seconds.
>>
>>> If I walk down the call stack, it returns from the _cursor.execute
>>> call in cursor.py and then hangs right before it goes into 
>>> _wait_timeout.
>>>
>>
> Hmm, yeah - my debugger is doing some weird off-by-one thing. If I put
> the break point on the while 1, it blocks there! If I put it on the poll()
> call, then it does do as you describe.
>
 It's a psycopg2 call, it should return the specific status but it
 doesn't, so right now I am digging the Psycopg2 code.

>>>
>>> If I remove RAISE statements and then execute SELECT 1; SELECT
>>> fails_after(30); it works as expected.
>>>
>>> As per the documentation,  (Ref:
>>> http://initd.org/psycopg/docs/connection.html)
>>> Connection.poll returns one of the constants defined in Poll constants
>>> . If it
>>> returns POLL_OK
>>>  
>>> then
>>> the connection has been established or the query results are available on
>>> the client. Otherwise wait until the file descriptor returned by
>>> fileno()
>>>  is
>>> ready to read or to write.
>>>
>>> So, in case of Raise no file descriptor to read or write, and so before
>>> raising the notice, It will block the connection till the pg_sleep
>>> completes.
>>>
>>
>> Thanks - confirmed, and created a test case for a bug report:
>> https://github.com/psycopg/psycopg2/issues/856
>>
>>
> This issue has been resolved in psycopg2 v2.8.
> @Dave, can you please move it to *in testing* as I couldn't.
>
>> --
>> Dave Page
>> Blog: http://pgsnake.blogspot.com
>> Twitter: @pgsnake
>>
>> EnterpriseDB UK: http://www.enterprisedb.com
>> The Enterprise PostgreSQL Company
>>
>

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Async queries pretending to be synchronous

[Khushboo Vashi]

On Tue, Mar 5, 2019 at 10:13 PM Dave Page  wrote:

>
>
> On Tue, Mar 5, 2019 at 6:36 AM Khushboo Vashi <
> khushboo.va...@enterprisedb.com> wrote:
>
>>
>>
>> On Mon, Mar 4, 2019 at 4:05 PM Khushboo Vashi <
>> khushboo.va...@enterprisedb.com> wrote:
>>
>>>
>>>
>>> On Mon, Mar 4, 2019 at 4:00 PM Dave Page  wrote:
>>>


 On Mon, Mar 4, 2019 at 10:18 AM Khushboo Vashi <
 khushboo.va...@enterprisedb.com> wrote:

>
>
> On Mon, Mar 4, 2019 at 3:43 PM Dave Page  wrote:
>
>> Hi
>>
>> On Mon, Mar 4, 2019 at 5:43 AM Khushboo Vashi <
>> khushboo.va...@enterprisedb.com> wrote:
>>
>>>
>>>
>>> On Fri, Mar 1, 2019 at 10:01 PM Dave Page  wrote:
>>>
 In investigating #3656 I found the initial problem to be that when
 running in a container, Gunicorn will kill the worker process if a 
 thread
 doesn't respond for 30 seconds by default. I fixed that by making the
 timeout match the application session timeout, but it revealed another
 issue.

 Given the function below (from the ticket), if you open the query
 tool and run:

 SELECT 1; SELECT fails_after(30);

 the async query actually blocks for 30 seconds in the cur.execute()
 call in execute_async() in connection.py (line 968). This causes the 
 entire
 app to hang (watch the dashboard requests pile up in pending state in 
 the
 network tab of the browser dev tools).

 If you run just the second SELECT, it works as expected, as does
 running something like:

 SELECT 1; SELECT pg_sleep(30);

 Anyone have any idea what's going on?

>>>
>>> Connection.poll() blocking the call here. ( connection.py -  
>>> _wait_timeout
>>> function line #1378 state = conn.poll() )
>>> In the asynchronous connection, after executing the query, the
>>> conn.poll() is being called to fetch the connection status.
>>> It gives the status accordingly but in this case, it is blocking and
>>> not giving the status.
>>>
>>
>> If I put a breakpoint on the _wait_timeout call (line 969 in
>> execute_async), it hits it *after* the 30 seconds has passed (during 
>> which
>> time all other queries for cancel or dashboards and status etc don't get
>> processed).
>>
>> Put the breakpoint  in the _wait_timeout function itself. The line no
> 1378  (state = conn.poll()) is blocking the execution 30 seconds.
>
>> If I walk down the call stack, it returns from the _cursor.execute
>> call in cursor.py and then hangs right before it goes into _wait_timeout.
>>
>
 Hmm, yeah - my debugger is doing some weird off-by-one thing. If I put
 the break point on the while 1, it blocks there! If I put it on the poll()
 call, then it does do as you describe.

>>> It's a psycopg2 call, it should return the specific status but it
>>> doesn't, so right now I am digging the Psycopg2 code.
>>>
>>
>> If I remove RAISE statements and then execute SELECT 1; SELECT
>> fails_after(30); it works as expected.
>>
>> As per the documentation,  (Ref:
>> http://initd.org/psycopg/docs/connection.html)
>> Connection.poll returns one of the constants defined in Poll constants
>> . If it
>> returns POLL_OK
>>  
>> then
>> the connection has been established or the query results are available on
>> the client. Otherwise wait until the file descriptor returned by fileno()
>>  is
>> ready to read or to write.
>>
>> So, in case of Raise no file descriptor to read or write, and so before
>> raising the notice, It will block the connection till the pg_sleep
>> completes.
>>
>
> Thanks - confirmed, and created a test case for a bug report:
> https://github.com/psycopg/psycopg2/issues/856
>
>
This issue has been resolved in psycopg2 v2.8.
@Dave, can you please move it to *in testing* as I couldn't.

> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EnterpriseDB UK: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

[pgAdmin4][Patch]: Psycopg2 v2.8 fixes

[Khushboo Vashi]

Hi,

Please find the attached patch to make pgAdmin compatible with psycopg2 v
2.8.
I have also made sure, pgAdmin will work with the older version of psycopg2.

Thanks,
Khushboo


psycopg2_2.8_fixes.patch
Description: Binary data