If this problem related to rlimit_files (open files limit) do i need to change /etc/security/limits.conf for php-fpm user or for sphinx user ?
Also where is temp file located? I do not see it in piler_document_root/tmp/ 2014-08-28 13:36 GMT+03:00 Konstantin <[email protected]>: > > how would you pass 100000 id values to pilerexport? > > Like i did for reindex > > for ID in `cat /tmp/metadata.csv` ; do reindex -f $ID -t $ID ; done > > It cound be a sinle id or range. > > > > > 2014-08-28 13:24 GMT+03:00 Janos SUTO <[email protected]>: > > >> On 2014-08-28 12:12, Konstantin wrote: >> >>> i make search by body and subject, so pilerexport can not help me here. >>> >>> I can execute search directly in sphinx and find id of the message. >>> It would be good to use pilerexport to export by id. >>> >> >> how would you pass 100000 id values to pilerexport? >> >> >> >> I have following modifications >>> sprinx max_matches = 100000 >>> >>> piler iu $config['MAX_SEARCH_HITS'] = 100000; >>> >>> Will enable more verbose is php-fpm log and see what happen. >>> >> >> ok, be sure to monitor real time resource usage. >> >> >> Janos >> >> >> >> 2014-08-28 12:13 GMT+03:00 Janos SUTO <[email protected]>: >>> >>> On 2014-08-27 23:33, John Kenyon wrote: >>>> >>>> I can see why people would want to use the WEBUI to export email, >>>>> however Janos is right, for a huge amount of messages it is better to >>>>> use the pilerexport command. >>>>> >>>>> Janos, perhaps to bridge the gap we could look at a dedicated >>>>> pilerexport feature in the WEBUI? >>>>> >>>>> Maybe a pilerexport form which auditors/admins have access to, they >>>>> input variables for pilerexport, once submitted it then runs an export >>>>> in the background, upon completion piler emails them back with a link >>>>> to download the zip? >>>>> >>>>> This approach would allow people to batch exports, thoughts? >>>>> >>>> >>>> good idea, however pilerexport doesn't use sphinx, so you are limited to >>>> specifying the sender, recipient, size, and date. >>>> >>>> However it might be possible to overcome the sphinx limit using some >>>> sort >>>> of iteration, though. >>>> >>>> Janos >>>> >>>> Regards, >>>>> >>>>> John Kenyon >>>>> >>>>> -----Original Message----- >>>>>> From: Janos SUTO [mailto:[email protected]] >>>>>> Sent: Thursday, 28 August 2014 6:34 AM >>>>>> To: Piler User >>>>>> Subject: Re: Download all count >>>>>> >>>>>> Hello, >>>>>> >>>>>> recent versions of the gui show you the total number of emails the >>>>>> search >>>>>> terms can be found. However sphinx poses a limit on the returned >>>>>> hits, so >>>>>> you may get the following response in the middle status bar: >>>>>> >>>>>> 1-50, total: 1000 (27271) >>>>>> >>>>>> meaning that sphinx found 27271 messages matching the search query, >>>>>> however sphinx itself returned you 1000 hits only, because of the >>>>>> sphinx >>>>>> internal (though configurable) limits, see Q6 in the FAQ. >>>>>> >>>>>> Also note that downloading 40000 emails in a single zip file requires >>>>>> huge >>>>>> amount of resources, not to mention what if the http client times >>>>>> out, or the >>>>>> php interpreter decides to end the script for excessing time limits. >>>>>> >>>>>> Janos >>>>>> >>>>>> On 2014-08-27 18:46, Konstantin wrote: >>>>>> >>>>>> Hi >>>>>>> >>>>>>> When i make a search in my piler ui i see ~40 000 results for my >>>>>>> request. >>>>>>> I used 'download all' button to download them in zip file. >>>>>>> >>>>>>> The problem is that inside zip file i see only 5 000 - 6 000 eml >>>>>>> file. >>>>>>> Count is different for the same search request. The same problem for >>>>>>> another search. >>>>>>> piler 0.1.25-master-branch, build 869. >>>>>>> I do not see any errors in syslog, maillog, nginx log. >>>>>>> >>>>>>> Is there any limits or known issues? >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> THIS MESSAGE WAS DELIVERED USING 100% RECYCLED ELECTRONS. >>>>>>> >>>>>> >>>>>> !DSPAM:5,53fe40c1155447497421850! >>>>>> >>>>> >>> -- >>> >>> THIS MESSAGE WAS DELIVERED USING 100% RECYCLED ELECTRONS. >>> >> >> > > > -- > *This message was delivered using 100% recycled electrons*. > -- *This message was delivered using 100% recycled electrons*.
