RE: pilerimport stopped saving files?
Janos, Removing the rows from the metadata and rcpt tables and then reimporting appears to have done the trick. Thank you! --Jeremiah -Original Message- From: Janos SUTO Sent: Friday, May 5, 2023 12:55 To: Piler User Subject: Re: pilerimport stopped saving files? [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe. On 2023-05-05 18:18, Jeremiah Poling wrote: > > Is there a way to reimport the failed ones? > > Maybe somehow delete a date range of mails from the system and then > reimport that range? > > I still have access to the source eml files from the date range in > question, but trying to import them now just results in a bunch of > “duplicate: XXX” messages in the console that's a tricky one. As I mentioned, piler discard these emails, because they already exist in the metadata table. You have 2 options: a) the "cannot open()" issue indicated that the given file is not present in the store directory structure. So option #1 is to get your backup, and restore the missing files b) if you don't have backups, then try removing the affected emails from both the metadata and rcpt tables. Also you have the attachment table which stores info to reassemble emails with attachments. Since an attachment may be present in several emails, I'm not sure if you should remove any rows from the attachment table. Anyway, be sure to backup the piler mysql database. Janos > From: Janos SUTO > Sent: Thursday, May 4, 2023 23:03 > To: Piler User > Subject: Re: pilerimport stopped saving files? > > [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you > recognize the sender and know the content is safe. > > Hello Jeremiah, > > if you have newer emails to import, then run pilerimport from the > command line, and look for errors both on stdin and in the mail logs. > If no errors then try to retrieve those emails with pilerget. > > Janos SUTO > > On May 4, 2023, at 22:59, Jeremiah Poling wrote: > >> Janos, >> >> Sorry to pester you again, but I've run into an unusual issue that >> I'm struggling to troubleshoot. >> >> I'm using pilerimport to load eml files that have been fetched using >> offlineimap. >> This was working fine until February 22 of this year. >> I don't remember having done anything that day, but I assume I must >> have run an update or something that broke it. >> >> Since then, the imported mails are being loaded in to the mysql >> database and sphinx (they show up in the web interface) but they're >> not being saved to the filesystem. >> so they can't be found by the web interface to preview: " Message >> Failed Verification" >> and they can't be loaded by pilerget: pilerget[3417]: >> > /var/piler/store/00/645/a4/cb/500064540b9d1979a504001a4529a4cb.m: >> cannot open() >> >> any thoughts on how I can troubleshoot this? >> >> Thanks, >> Jeremiah
Re: pilerimport stopped saving files?
On 2023-05-05 18:18, Jeremiah Poling wrote: Is there a way to reimport the failed ones? Maybe somehow delete a date range of mails from the system and then reimport that range? I still have access to the source eml files from the date range in question, but trying to import them now just results in a bunch of “duplicate: XXX” messages in the console that's a tricky one. As I mentioned, piler discard these emails, because they already exist in the metadata table. You have 2 options: a) the "cannot open()" issue indicated that the given file is not present in the store directory structure. So option #1 is to get your backup, and restore the missing files b) if you don't have backups, then try removing the affected emails from both the metadata and rcpt tables. Also you have the attachment table which stores info to reassemble emails with attachments. Since an attachment may be present in several emails, I'm not sure if you should remove any rows from the attachment table. Anyway, be sure to backup the piler mysql database. Janos From: Janos SUTO Sent: Thursday, May 4, 2023 23:03 To: Piler User Subject: Re: pilerimport stopped saving files? [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe. Hello Jeremiah, if you have newer emails to import, then run pilerimport from the command line, and look for errors both on stdin and in the mail logs. If no errors then try to retrieve those emails with pilerget. Janos SUTO On May 4, 2023, at 22:59, Jeremiah Poling wrote: Janos, Sorry to pester you again, but I've run into an unusual issue that I'm struggling to troubleshoot. I'm using pilerimport to load eml files that have been fetched using offlineimap. This was working fine until February 22 of this year. I don't remember having done anything that day, but I assume I must have run an update or something that broke it. Since then, the imported mails are being loaded in to the mysql database and sphinx (they show up in the web interface) but they're not being saved to the filesystem. so they can't be found by the web interface to preview: " Message Failed Verification" and they can't be loaded by pilerget: pilerget[3417]: /var/piler/store/00/645/a4/cb/500064540b9d1979a504001a4529a4cb.m: cannot open() any thoughts on how I can troubleshoot this? Thanks, Jeremiah
RE: pilerimport stopped saving files?
I went to try and import a few new messages this morning to see what the logs would say. When checking those, there were no unusual errors, and then I noticed that it seems to have been working correctly for new messages from the past couple of days. I rebooted the VM this is running on as part of my original troubleshooting, so perhaps that fixed whatever had gone haywire. However, I’m still left with a range of emails from Feb yesterday that are broken/missing in the archive. Is there a way to reimport the failed ones? Maybe somehow delete a date range of mails from the system and then reimport that range? I still have access to the source eml files from the date range in question, but trying to import them now just results in a bunch of “duplicate: XXX” messages in the console Thanks, Jeremiah From: Janos SUTO Sent: Thursday, May 4, 2023 23:03 To: Piler User Subject: Re: pilerimport stopped saving files? [EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe. Hello Jeremiah, if you have newer emails to import, then run pilerimport from the command line, and look for errors both on stdin and in the mail logs. If no errors then try to retrieve those emails with pilerget. Janos SUTO On May 4, 2023, at 22:59, Jeremiah Poling mailto:jpol...@anra.org>> wrote: Janos, Sorry to pester you again, but I've run into an unusual issue that I'm struggling to troubleshoot. I'm using pilerimport to load eml files that have been fetched using offlineimap. This was working fine until February 22 of this year. I don't remember having done anything that day, but I assume I must have run an update or something that broke it. Since then, the imported mails are being loaded in to the mysql database and sphinx (they show up in the web interface) but they're not being saved to the filesystem. so they can't be found by the web interface to preview: " Message Failed Verification" and they can't be loaded by pilerget: pilerget[3417]: /var/piler/store/00/645/a4/cb/500064540b9d1979a504001a4529a4cb.m: cannot open() any thoughts on how I can troubleshoot this? Thanks, Jeremiah
Re: pilerimport stopped saving files?
Hello Jeremiah, if you have newer emails to import, then run pilerimport from the command line, and look for errors both on stdin and in the mail logs. If no errors then try to retrieve those emails with pilerget. Janos SUTO On May 4, 2023, 22:59, at 22:59, Jeremiah Poling wrote: > > >Janos, > >Sorry to pester you again, but I've run into an unusual issue that I'm >struggling to troubleshoot. > >I'm using pilerimport to load eml files that have been fetched using >offlineimap. >This was working fine until February 22 of this year. >I don't remember having done anything that day, but I assume I must >have run an update or something that broke it. > >Since then, the imported mails are being loaded in to the mysql >database and sphinx (they show up in the web interface) but they're not >being saved to the filesystem. >so they can't be found by the web interface to preview: " Message >Failed Verification" >and they can't be loaded by pilerget: pilerget[3417]: >/var/piler/store/00/645/a4/cb/500064540b9d1979a504001a4529a4cb.m: >cannot open() > >any thoughts on how I can troubleshoot this? > >Thanks, >Jeremiah
pilerimport stopped saving files?
Janos, Sorry to pester you again, but I've run into an unusual issue that I'm struggling to troubleshoot. I'm using pilerimport to load eml files that have been fetched using offlineimap. This was working fine until February 22 of this year. I don't remember having done anything that day, but I assume I must have run an update or something that broke it. Since then, the imported mails are being loaded in to the mysql database and sphinx (they show up in the web interface) but they're not being saved to the filesystem. so they can't be found by the web interface to preview: " Message Failed Verification" and they can't be loaded by pilerget: pilerget[3417]: /var/piler/store/00/645/a4/cb/500064540b9d1979a504001a4529a4cb.m: cannot open() any thoughts on how I can troubleshoot this? Thanks, Jeremiah
