> how would you pass 100000 id values to pilerexport? Like i did for reindex
for ID in `cat /tmp/metadata.csv` ; do reindex -f $ID -t $ID ; done It cound be a sinle id or range. 2014-08-28 13:24 GMT+03:00 Janos SUTO <[email protected]>: > > On 2014-08-28 12:12, Konstantin wrote: > >> i make search by body and subject, so pilerexport can not help me here. >> >> I can execute search directly in sphinx and find id of the message. >> It would be good to use pilerexport to export by id. >> > > how would you pass 100000 id values to pilerexport? > > > > I have following modifications >> sprinx max_matches = 100000 >> >> piler iu $config['MAX_SEARCH_HITS'] = 100000; >> >> Will enable more verbose is php-fpm log and see what happen. >> > > ok, be sure to monitor real time resource usage. > > > Janos > > > > 2014-08-28 12:13 GMT+03:00 Janos SUTO <[email protected]>: >> >> On 2014-08-27 23:33, John Kenyon wrote: >>> >>> I can see why people would want to use the WEBUI to export email, >>>> however Janos is right, for a huge amount of messages it is better to >>>> use the pilerexport command. >>>> >>>> Janos, perhaps to bridge the gap we could look at a dedicated >>>> pilerexport feature in the WEBUI? >>>> >>>> Maybe a pilerexport form which auditors/admins have access to, they >>>> input variables for pilerexport, once submitted it then runs an export >>>> in the background, upon completion piler emails them back with a link >>>> to download the zip? >>>> >>>> This approach would allow people to batch exports, thoughts? >>>> >>> >>> good idea, however pilerexport doesn't use sphinx, so you are limited to >>> specifying the sender, recipient, size, and date. >>> >>> However it might be possible to overcome the sphinx limit using some sort >>> of iteration, though. >>> >>> Janos >>> >>> Regards, >>>> >>>> John Kenyon >>>> >>>> -----Original Message----- >>>>> From: Janos SUTO [mailto:[email protected]] >>>>> Sent: Thursday, 28 August 2014 6:34 AM >>>>> To: Piler User >>>>> Subject: Re: Download all count >>>>> >>>>> Hello, >>>>> >>>>> recent versions of the gui show you the total number of emails the >>>>> search >>>>> terms can be found. However sphinx poses a limit on the returned hits, >>>>> so >>>>> you may get the following response in the middle status bar: >>>>> >>>>> 1-50, total: 1000 (27271) >>>>> >>>>> meaning that sphinx found 27271 messages matching the search query, >>>>> however sphinx itself returned you 1000 hits only, because of the >>>>> sphinx >>>>> internal (though configurable) limits, see Q6 in the FAQ. >>>>> >>>>> Also note that downloading 40000 emails in a single zip file requires >>>>> huge >>>>> amount of resources, not to mention what if the http client times out, >>>>> or the >>>>> php interpreter decides to end the script for excessing time limits. >>>>> >>>>> Janos >>>>> >>>>> On 2014-08-27 18:46, Konstantin wrote: >>>>> >>>>> Hi >>>>>> >>>>>> When i make a search in my piler ui i see ~40 000 results for my >>>>>> request. >>>>>> I used 'download all' button to download them in zip file. >>>>>> >>>>>> The problem is that inside zip file i see only 5 000 - 6 000 eml file. >>>>>> Count is different for the same search request. The same problem for >>>>>> another search. >>>>>> piler 0.1.25-master-branch, build 869. >>>>>> I do not see any errors in syslog, maillog, nginx log. >>>>>> >>>>>> Is there any limits or known issues? >>>>>> >>>>>> Thanks. >>>>>> >>>>>> -- >>>>>> >>>>>> THIS MESSAGE WAS DELIVERED USING 100% RECYCLED ELECTRONS. >>>>>> >>>>> >>>>> !DSPAM:5,53fe40c1155447497421850! >>>>> >>>> >> -- >> >> THIS MESSAGE WAS DELIVERED USING 100% RECYCLED ELECTRONS. >> > > -- *This message was delivered using 100% recycled electrons*.
