Re: Piler 1.3.10 issues with imap importing (Lotus Domino Server) from GUI

[sj]

Hello Hugo,

I'm so sorry that the fix took quite some time to be made. Please
get this commit 
https://bitbucket.org/jsuto/piler/commits/8685574ede416cfce65fb8b1ebc950764778c898

and it solves this issue.

Janos


On 2021-01-27 16:35, Hugo Alvarez wrote:

Hi Janos,

First thanks for all.

Regarding the python version, sorry for the mistake, we are indeed
using python3 (3.6).

Regarding the problem with the import script, the process is actually
finished correctly, the issue is that the relevant fields in the
'import' database table does not seem to get updated.

Upon creation of the task through the gui, the first run of the
import.sh script will actually import all the emails, but only the
following fields get updated in the database:

status: 2 (from imapfetch.py and then from import.c)

total: abc (from imapfetch.py)
imported: xyz (from import.c)

The problem here is that when imapfech.py runs again (when cron reruns
import.sh), the executed query is:

SELECT id, server, username, password FROM import WHERE started=0

which in turn will return all the import jobs again.

Hope this makes the problem clear.

Thanks

Regards

 [1]
Libre de virus. www.avast.com [1]

El mar, 26 ene 2021 a las 20:34,  escribió:


Hello Hugo,

thank you for the pacthes. Have you tried to use imapfetch.py with
python3?
Also can you track what happens or at what point pilerimport stucks?

Anything
relevant in the mail logs?

I've just re-run the import test, and pilerimport has properly
updated
the
import table.

Janos

On 2021-01-25 13:27, Hugo Alvarez wrote:


We have installed Piler 1.3.10 and we cannot get imap importing

(Lotus

Domino server) from GUI. Here is what we have been able to

trackdown

so far:

1 - In read_folder_list we had to decode to utf-8  the case for

the

tupple type. A type missmatch error would occur otherwise:

elif isinstance(folder, type(())):
folder = re.sub(r'\{\d+\}$', '',
folder[0].decode('utf-8')) + folder[1].decode('utf-8')

2 - Domino's Imap server returns a backslash the field separator:

[b'(\\Noinferiors \\HasNoChildren) "" Inbox', b'(\\Noinferiors
\\HasNoChildren) "" Trash', b'(\\HasNoChildren) ""

Drafts',

b'(\\HasNoChildren) "" Sent']

Which does not match against the exiting regexp in

read_folder_list in

imapfetch.py. This is fixed adding the backslash to the regexp:  f

=

re.split(r' \"[\/\.\\]+\" ', folder)

3 - We are using python 2.7.5 under Centos 7.9.2009 which seens to
have a bug in imaplib's _checkquote(), which prevents folder names
with special characters from being correctly escaped making
imapfetch.py to fail. We fixed this by using a  sane _checkquote()
version from a previous imaplib. Maybe there is a cleaner solution

...


result.append(_checkquote(conn, f[1]))

4 - In process_folder, empty emails will fail when trying to write

to

disk. This is fixed by checking for a tupple:

for num in data[0].split():
rc, data = conn.fetch(num, '(RFC822)')
if opts['verbose']:
print(rc, num)
if isinstance(data[0], tuple):
opts['counter'] += 1
with open("{}.eml".format(opts['counter']), "wb")

as

f:f.write(data[0][1])

We could make it work with these changes, but pilerimport -T does

not

seem to mark the fields 'updated' or 'finished' in the import

table,

so subsequent executions of import.sh starts the whole process

again

(failing the import due to duplicates).




Links:
--
[1] 
https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail

Re: Piler 1.3.10 issues with imap importing (Lotus Domino Server) from GUI

[Hugo Alvarez]

Hi Janos,

First thanks for all.

Regarding the python version, sorry for the mistake, we are indeed using
python3 (3.6).

Regarding the problem with the import script, the process is actually
finished correctly, the issue is that the relevant fields in
the 'import' database table does not seem to get updated.

Upon creation of the task through the gui, the first run of the import.sh
script will actually import all the emails, but only the following fields
get updated in the database:

status: 2 (from imapfetch.py and then from import.c)
total: abc (from imapfetch.py)
imported: xyz (from import.c)

The problem here is that when imapfech.py runs again (when cron reruns
import.sh), the executed query is:

SELECT id, server, username, password FROM import WHERE started=0

which in turn will return all the import jobs again.

Hope this makes the problem clear.

Thanks

Regards


Libre
de virus. www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

El mar, 26 ene 2021 a las 20:34,  escribió:

>
>
> Hello Hugo,
>
> thank you for the pacthes. Have you tried to use imapfetch.py with
> python3?
> Also can you track what happens or at what point pilerimport stucks?
> Anything
> relevant in the mail logs?
>
> I've just re-run the import test, and pilerimport has properly updated
> the
> import table.
>
> Janos
>
> On 2021-01-25 13:27, Hugo Alvarez wrote:
> >
> > We have installed Piler 1.3.10 and we cannot get imap importing (Lotus
> > Domino server) from GUI. Here is what we have been able to trackdown
> > so far:
> >
> > 1 - In read_folder_list we had to decode to utf-8  the case for the
> > tupple type. A type missmatch error would occur otherwise:
> >
> > elif isinstance(folder, type(())):
> > folder = re.sub(r'\{\d+\}$', '',
> > folder[0].decode('utf-8')) + folder[1].decode('utf-8')
> >
> > 2 - Domino's Imap server returns a backslash the field separator:
> >
> > [b'(\\Noinferiors \\HasNoChildren) "" Inbox', b'(\\Noinferiors
> > \\HasNoChildren) "" Trash', b'(\\HasNoChildren) "" Drafts',
> > b'(\\HasNoChildren) "" Sent']
> >
> > Which does not match against the exiting regexp in read_folder_list in
> > imapfetch.py. This is fixed adding the backslash to the regexp:  f =
> > re.split(r' \"[\/\.\\]+\" ', folder)
> >
> > 3 - We are using python 2.7.5 under Centos 7.9.2009 which seens to
> > have a bug in imaplib's _checkquote(), which prevents folder names
> > with special characters from being correctly escaped making
> > imapfetch.py to fail. We fixed this by using a  sane _checkquote()
> > version from a previous imaplib. Maybe there is a cleaner solution ...
> >
> > result.append(_checkquote(conn, f[1]))
> >
> > 4 - In process_folder, empty emails will fail when trying to write to
> > disk. This is fixed by checking for a tupple:
> >
> > for num in data[0].split():
> > rc, data = conn.fetch(num, '(RFC822)')
> > if opts['verbose']:
> > print(rc, num)
> > if isinstance(data[0], tuple):
> > opts['counter'] += 1
> > with open("{}.eml".format(opts['counter']), "wb") as
> > f:f.write(data[0][1])
> >
> > We could make it work with these changes, but pilerimport -T does not
> > seem to mark the fields 'updated' or 'finished' in the import table,
> > so subsequent executions of import.sh starts the whole process again
> > (failing the import due to duplicates).
> >
>
>

Re: Piler 1.3.10 issues with imap importing (Lotus Domino Server) from GUI

[sj]

Hello Hugo,

thank you for the pacthes. Have you tried to use imapfetch.py with 
python3?
Also can you track what happens or at what point pilerimport stucks? 
Anything

relevant in the mail logs?

I've just re-run the import test, and pilerimport has properly updated 
the

import table.

Janos

On 2021-01-25 13:27, Hugo Alvarez wrote:


We have installed Piler 1.3.10 and we cannot get imap importing (Lotus
Domino server) from GUI. Here is what we have been able to trackdown
so far:

1 - In read_folder_list we had to decode to utf-8  the case for the
tupple type. A type missmatch error would occur otherwise:

elif isinstance(folder, type(())):
folder = re.sub(r'\{\d+\}$', '',
folder[0].decode('utf-8')) + folder[1].decode('utf-8')

2 - Domino's Imap server returns a backslash the field separator:

[b'(\\Noinferiors \\HasNoChildren) "" Inbox', b'(\\Noinferiors
\\HasNoChildren) "" Trash', b'(\\HasNoChildren) "" Drafts',
b'(\\HasNoChildren) "" Sent']

Which does not match against the exiting regexp in read_folder_list in
imapfetch.py. This is fixed adding the backslash to the regexp:  f =
re.split(r' \"[\/\.\\]+\" ', folder)

3 - We are using python 2.7.5 under Centos 7.9.2009 which seens to
have a bug in imaplib's _checkquote(), which prevents folder names
with special characters from being correctly escaped making
imapfetch.py to fail. We fixed this by using a  sane _checkquote()
version from a previous imaplib. Maybe there is a cleaner solution ...

result.append(_checkquote(conn, f[1]))

4 - In process_folder, empty emails will fail when trying to write to
disk. This is fixed by checking for a tupple:

for num in data[0].split():
rc, data = conn.fetch(num, '(RFC822)')
if opts['verbose']:
print(rc, num)
if isinstance(data[0], tuple):
opts['counter'] += 1
with open("{}.eml".format(opts['counter']), "wb") as
f:f.write(data[0][1])

We could make it work with these changes, but pilerimport -T does not
seem to mark the fields 'updated' or 'finished' in the import table,
so subsequent executions of import.sh starts the whole process again
(failing the import due to duplicates).

Piler 1.3.10 issues with imap importing (Lotus Domino Server) from GUI

[Hugo Alvarez]

Hello,

We have installed Piler 1.3.10 and we cannot get imap importing (Lotus
Domino server) from GUI. Here is what we have been able to trackdown so far:

1 - In read_folder_list we had to decode to utf-8  the case for the tupple
type. A type missmatch error would occur otherwise:

elif isinstance(folder, type(())):
folder = re.sub(r'\{\d+\}$', '', folder[0].decode('utf-8')) +
folder[1].decode('utf-8')

2 - Domino's Imap server returns a backslash the field separator:

[b'(\\Noinferiors \\HasNoChildren) "" Inbox', b'(\\Noinferiors
\\HasNoChildren) "" Trash', b'(\\HasNoChildren) "" Drafts',
b'(\\HasNoChildren) "" Sent']

Which does not match against the exiting regexp in read_folder_list in
imapfetch.py. This is fixed adding the backslash to the regexp:  f =
re.split(r' \"[\/\.\\]+\" ', folder)

3 - We are using python 2.7.5 under Centos 7.9.2009 which seens to have a
bug in imaplib's _checkquote(), which prevents folder names with special
characters from being correctly escaped making imapfetch.py to fail. We
fixed this by using a  sane _checkquote() version from a previous imaplib.
Maybe there is a cleaner solution ...

result.append(_checkquote(conn, f[1]))

4 - In process_folder, empty emails will fail when trying to write to disk.
This is fixed by checking for a tupple:

for num in data[0].split():
rc, data = conn.fetch(num, '(RFC822)')
if opts['verbose']:
print(rc, num)
if isinstance(data[0], tuple):
opts['counter'] += 1
with open("{}.eml".format(opts['counter']), "wb") as f:
f.write(data[0][1])

We could make it work with these changes, but pilerimport -T does not seem
to mark the fields 'updated' or 'finished' in the import table, so
subsequent executions of import.sh starts the whole process again (failing
the import due to duplicates).

Hope this helps to fix the issues.

Regards


Libre
de virus. www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>