Re: pilerexport w flag matching
Hello Ryan, try getting the master branch, do a complete build, however, don't install it. Instead enter the src directory, use LD_LIBRARY_PATH=. to make sure ./pilerexport will use the recently built libpiler.so file, and try again. Also it's worth to check what's going on 127.0.0.1:9306 with ngrep or similar tool. Janos On 2021-04-08 08:45, Ryan Blenis wrote: Hi Janos, Thanks as always for the reply. No worries on the -w replacing the other switches, a documentation update will clear that up pretty easily, and thank you, I'd love a total in the dry run! As for "the export utility assumes searchd is listening on 127.0.0.1:9306 [1] That part didn't change from 1.3.8 to 1.3.11." I looked back at all the commits and didn't see anything in that file that could be causing it, but I'll investigate because now it's just annoying me that it doesn't work. I saw that it was assuming searchd was on 127.0.0.1:9306 [1], and it is on my system, however it still fails with that error in the new build. I don't know if you have a test case for pilerexport -w, but it should be repeatable. I don't see much changing in the cfg.c either, but when I printf the cfg.mysqluser and cfg.mysqlpwd on the old (1.3.8) version I get the correct info. If I printf cfg.mysqluser and cfg.mysqlpwd on the new version I get "r" and "" respectively. A quick strace of pilerexport shows it opening the correct conf file (/usr/local/etc/piler/piler.conf) on both versions. And specifying the config file with -c did not help either. Not sure if you can reproduce this on your end or not. The odd part is, cfg.mysqluser and cfg.mysqlpwd _IS_ utilized in the "init_session_data(&sdata, &cfg);" line above the "init_session_data(&sdata2, &cfg);" and it passes the database "open" test just fine there... Thank you. On Thu, Apr 8, 2021 at 12:13 AM wrote: Hello Ryan, On 2021-04-08 01:36, Ryan Blenis wrote: Thanks, that led me to what is causing the issue / confusion. The -w switch is described as "Where condition to pass to sphinx, eg. "match('@subject: piler')" Which led me to believe the MATCH string was all that was supposed to be there/replaced, however a quick look at the code shows that if -w is used, it REPLACES the ENTIRE where clause. This distinction means yes, perhaps the "eg." was not that prominent in the short --help output. I'll improve the docs on the website, it's lagging behind the actual features, and I'll add a clarification on it. The simplest workaround to this for others would be to note that -w allows you to build your own query and negates the use of other parameters. The ideal fix I think would be to still utilize the other parameters, but have -w content appended within the MATCH() portion of the query. Such fix would only complicate things because you can define the whole query using -w including the time frame, recipients, etc. Again, I'll add a clarification to the docs. Aside from that: I realize I'm behind on piler (1.3.8), and would like to update to get the latest pilerexport with zip capabilities, yet I see there is no upgrade information on https://www.mailpiler.org/wiki/current:upgrade . What is the process to the latest (1.3.11)? Well, simply compile the new stuff, and overwrite the binaries and the GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11. However, don't rush with that. The zip export feature has a poor performance that needs a rework. I'd also like to add a "--num-only" type flag to pilerexport to see the number of matches before exporting (would probably imply dryrun). We have something similar. When specifying -d (or --dry-run) it prints the matching serial ids, eg. $ pilerexport -d -w "MATCH(' some query')" id:318 id:375 id:518 id:656 id:660 id:688 id:733 I'll improve it to add "total:7" as the last line of the output, if that's OK. When trying to just compile the latest, I get the error "error: cannot connect to 127.0.0.1:9306 [1] [1]" so I'm not sure if that's an issue because not all the components are upgraded, or if I had a different configure flag/path configured during the original install. The export utility assumes searchd is listening on 127.0.0.1:9306 [1] That part didn't change from 1.3.8 to 1.3.11. Janos Links: -- [1] http://127.0.0.1:9306
Re: pilerexport w flag matching
Hi Janos, Thanks as always for the reply. No worries on the -w replacing the other switches, a documentation update will clear that up pretty easily, and thank you, I'd love a total in the dry run! As for "the export utility assumes searchd is listening on 127.0.0.1:9306 That part didn't change from 1.3.8 to 1.3.11." I looked back at all the commits and didn't see anything in that file that could be causing it, but I'll investigate because now it's just annoying me that it doesn't work. I saw that it was assuming searchd was on 127.0.0.1:9306, and it is on my system, however it still fails with that error in the new build. I don't know if you have a test case for pilerexport -w, but it should be repeatable. I don't see much changing in the cfg.c either, but when I printf the cfg.mysqluser and cfg.mysqlpwd on the old (1.3.8) version I get the correct info. If I printf cfg.mysqluser and cfg.mysqlpwd on the new version I get "r" and "" respectively. A quick strace of pilerexport shows it opening the correct conf file (/usr/local/etc/piler/piler.conf) on both versions. And specifying the config file with -c did not help either. Not sure if you can reproduce this on your end or not. The odd part is, cfg.mysqluser and cfg.mysqlpwd _IS_ utilized in the "init_session_data(&sdata, &cfg);" line above the "init_session_data(&sdata2, &cfg);" and it passes the database "open" test just fine there... Thank you. On Thu, Apr 8, 2021 at 12:13 AM wrote: > > > Hello Ryan, > > On 2021-04-08 01:36, Ryan Blenis wrote: > > > > Thanks, that led me to what is causing the issue / confusion. > > > > The -w switch is described as "Where condition to pass to sphinx, eg. > > "match('@subject: piler')" > > > > Which led me to believe the MATCH string was all that was supposed to > > be there/replaced, however a quick look at the code shows that if -w > > is used, it REPLACES the ENTIRE where clause. This distinction means > > yes, perhaps the "eg." was not that prominent in the short --help > output. > I'll improve the docs on the website, it's lagging behind the actual > features, > and I'll add a clarification on it. > > > The simplest workaround to this for others would be to note that -w > > allows you to build your own query and negates the use of other > > parameters. The ideal fix I think would be to still utilize the other > > parameters, but have -w content appended within the MATCH() portion of > > the query. > > Such fix would only complicate things because you can define the whole > query using -w including the time frame, recipients, etc. Again, I'll > add a clarification to the docs. > > > Aside from that: I realize I'm behind on piler (1.3.8), and would like > > to update to get the latest pilerexport with zip capabilities, yet I > > see there is no upgrade information on > > https://www.mailpiler.org/wiki/current:upgrade . What is the process > > to the latest (1.3.11)? > > Well, simply compile the new stuff, and overwrite the binaries and the > GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11. > However, don't rush with that. The zip export feature has a poor > performance that needs a rework. > > > I'd also like to add a "--num-only" type flag to pilerexport to see > > the number of matches before exporting (would probably imply dryrun). > > We have something similar. When specifying -d (or --dry-run) it prints > the matching serial ids, eg. > > $ pilerexport -d -w "MATCH(' some query')" > id:318 > id:375 > id:518 > id:656 > id:660 > id:688 > id:733 > > I'll improve it to add "total:7" as the last line of the output, if > that's OK. > > > When trying to just compile the latest, I get the error "error: cannot > > connect to 127.0.0.1:9306 [1]" so I'm not sure if that's an issue > > because not all the components are upgraded, or if I had a different > > configure flag/path configured during the original install. > > The export utility assumes searchd is listening on 127.0.0.1:9306 > That part didn't change from 1.3.8 to 1.3.11. > > Janos > >
Re: pilerexport w flag matching
Hello Ryan, On 2021-04-08 01:36, Ryan Blenis wrote: Thanks, that led me to what is causing the issue / confusion. The -w switch is described as "Where condition to pass to sphinx, eg. "match('@subject: piler')" Which led me to believe the MATCH string was all that was supposed to be there/replaced, however a quick look at the code shows that if -w is used, it REPLACES the ENTIRE where clause. This distinction means yes, perhaps the "eg." was not that prominent in the short --help output. I'll improve the docs on the website, it's lagging behind the actual features, and I'll add a clarification on it. The simplest workaround to this for others would be to note that -w allows you to build your own query and negates the use of other parameters. The ideal fix I think would be to still utilize the other parameters, but have -w content appended within the MATCH() portion of the query. Such fix would only complicate things because you can define the whole query using -w including the time frame, recipients, etc. Again, I'll add a clarification to the docs. Aside from that: I realize I'm behind on piler (1.3.8), and would like to update to get the latest pilerexport with zip capabilities, yet I see there is no upgrade information on https://www.mailpiler.org/wiki/current:upgrade . What is the process to the latest (1.3.11)? Well, simply compile the new stuff, and overwrite the binaries and the GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11. However, don't rush with that. The zip export feature has a poor performance that needs a rework. I'd also like to add a "--num-only" type flag to pilerexport to see the number of matches before exporting (would probably imply dryrun). We have something similar. When specifying -d (or --dry-run) it prints the matching serial ids, eg. $ pilerexport -d -w "MATCH(' some query')" id:318 id:375 id:518 id:656 id:660 id:688 id:733 I'll improve it to add "total:7" as the last line of the output, if that's OK. When trying to just compile the latest, I get the error "error: cannot connect to 127.0.0.1:9306 [1]" so I'm not sure if that's an issue because not all the components are upgraded, or if I had a different configure flag/path configured during the original install. The export utility assumes searchd is listening on 127.0.0.1:9306 That part didn't change from 1.3.8 to 1.3.11. Janos
Re: pilerexport w flag matching
I should clarify the "error: cannot connect to 127.0.0.1:9306" error message. This error does not occur at compile time, but only at runtime of the latest pilerexport, and only when the -w switch is used. On Wed, Apr 7, 2021 at 7:36 PM Ryan Blenis wrote: > Hi Janos, > > Thanks, that led me to what is causing the issue / confusion. > > The -w switch is described as "Where condition to pass to sphinx, eg. > "match('@subject: piler')" > > Which led me to believe the MATCH string was all that was supposed to be > there/replaced, however a quick look at the code shows that if -w is used, > it REPLACES the ENTIRE where clause. This distinction means that the use of > -w negates the use of the a, b, and r switches as those parameters no > longer go through your query builder. I was getting more results because it > wasn't limited to a timeframe or to a recipient due to those flags being > "skipped". > > The simplest workaround to this for others would be to note that -w allows > you to build your own query and negates the use of other parameters. The > ideal fix I think would be to still utilize the other parameters, but have > -w content appended within the MATCH() portion of the query. > > Aside from that: I realize I'm behind on piler (1.3.8), and would like to > update to get the latest pilerexport with zip capabilities, yet I see there > is no upgrade information on > https://www.mailpiler.org/wiki/current:upgrade . What is the process to > the latest (1.3.11)? > > I'd also like to add a "--num-only" type flag to pilerexport to see the > number of matches before exporting (would probably imply dryrun). I didn't > see a way to do something like that already. If that's something that could > be added, great, if not, I'll try my hand at it and submit a patch. > > When trying to just compile the latest, I get the error "error: cannot > connect to 127.0.0.1:9306" so I'm not sure if that's an issue because not > all the components are upgraded, or if I had a different configure > flag/path configured during the original install. > > > On Wed, Apr 7, 2021 at 5:19 PM Ryan Blenis wrote: > >> Disregard that last email. Coffee is good, not re-running ./configure >> after installing deps is bad. Following up shortly with more pertinent >> info. Thank you. >> >> On Wed, Apr 7, 2021 at 3:58 PM Ryan Blenis wrote: >> >>> Hi Janos, >>> >>> Thanks for the response, in trying to do this (I cloned the repo, >>> ./configure --localstatedir=/var --with-database=mariadb , and ran make) >>> and got this: >>> >>> Making all in src >>> make[1]: Entering directory '/tmp/piler/src/piler/src' >>> gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2 >>> -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb >>> -I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o >>> pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre >>> -L/usr/lib/x86_64-linux-gnu/ -lmariadb -L. >>> /usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file': >>> /tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to >>> `zip_open' >>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined >>> reference to `zip_source_file' >>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined >>> reference to `zip_file_add' >>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined >>> reference to `zip_close' >>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined >>> reference to `zip_strerror' >>> collect2: error: ld returned 1 exit status >>> make[1]: *** [Makefile:63: pilerexport] Error 1 >>> make[1]: Leaving directory '/tmp/piler/src/piler/src' >>> make: *** [Makefile:41: all-recursive] Error 1 >>> >>> (Note that I originally got a zip.h not found error, which I ran apt >>> install libzip-dev. Ubuntu 20.04.2 LTS >>> >>> I can't seem to get past this point to recompile. >>> >>> >>> On Wed, Apr 7, 2021 at 2:50 PM wrote: >>> Hello Ryan, please apply this patch to pilerexport.c, and recompile it. https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d It syslogs the sphinx query. Then try the following. Enter the search query on the gui, and record the sphinx query syslogged. Then re-run the pilerexport command, and record the new sphinx query, and compare it with the previous value. Verify that even the single-quotes and double quotes are the same in both queries. Janos SUTO On 2021-04-07 18:18, Ryan Blenis wrote: > Hi Janos, > > I have to export potentially a ton of emails and was looking to use > pilerexport versus multiple batches of GUI searches. I saw the -w flag > and thought "great, I can use this" but it doesn't seem to respond > appropriately for my test case. I have 2 emails that match the > following (generalized terms used vs actual), limiting with -m 3 for > testing p
Re: pilerexport w flag matching
Hi Janos, Thanks, that led me to what is causing the issue / confusion. The -w switch is described as "Where condition to pass to sphinx, eg. "match('@subject: piler')" Which led me to believe the MATCH string was all that was supposed to be there/replaced, however a quick look at the code shows that if -w is used, it REPLACES the ENTIRE where clause. This distinction means that the use of -w negates the use of the a, b, and r switches as those parameters no longer go through your query builder. I was getting more results because it wasn't limited to a timeframe or to a recipient due to those flags being "skipped". The simplest workaround to this for others would be to note that -w allows you to build your own query and negates the use of other parameters. The ideal fix I think would be to still utilize the other parameters, but have -w content appended within the MATCH() portion of the query. Aside from that: I realize I'm behind on piler (1.3.8), and would like to update to get the latest pilerexport with zip capabilities, yet I see there is no upgrade information on https://www.mailpiler.org/wiki/current:upgrade . What is the process to the latest (1.3.11)? I'd also like to add a "--num-only" type flag to pilerexport to see the number of matches before exporting (would probably imply dryrun). I didn't see a way to do something like that already. If that's something that could be added, great, if not, I'll try my hand at it and submit a patch. When trying to just compile the latest, I get the error "error: cannot connect to 127.0.0.1:9306" so I'm not sure if that's an issue because not all the components are upgraded, or if I had a different configure flag/path configured during the original install. On Wed, Apr 7, 2021 at 5:19 PM Ryan Blenis wrote: > Disregard that last email. Coffee is good, not re-running ./configure > after installing deps is bad. Following up shortly with more pertinent > info. Thank you. > > On Wed, Apr 7, 2021 at 3:58 PM Ryan Blenis wrote: > >> Hi Janos, >> >> Thanks for the response, in trying to do this (I cloned the repo, >> ./configure --localstatedir=/var --with-database=mariadb , and ran make) >> and got this: >> >> Making all in src >> make[1]: Entering directory '/tmp/piler/src/piler/src' >> gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2 >> -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb >> -I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o >> pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre >> -L/usr/lib/x86_64-linux-gnu/ -lmariadb -L. >> /usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file': >> /tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to >> `zip_open' >> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined >> reference to `zip_source_file' >> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined >> reference to `zip_file_add' >> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined >> reference to `zip_close' >> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined >> reference to `zip_strerror' >> collect2: error: ld returned 1 exit status >> make[1]: *** [Makefile:63: pilerexport] Error 1 >> make[1]: Leaving directory '/tmp/piler/src/piler/src' >> make: *** [Makefile:41: all-recursive] Error 1 >> >> (Note that I originally got a zip.h not found error, which I ran apt >> install libzip-dev. Ubuntu 20.04.2 LTS >> >> I can't seem to get past this point to recompile. >> >> >> On Wed, Apr 7, 2021 at 2:50 PM wrote: >> >>> >>> Hello Ryan, >>> >>> please apply this patch to pilerexport.c, and recompile it. >>> >>> https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d >>> >>> It syslogs the sphinx query. Then try the following. Enter the search >>> query >>> on the gui, and record the sphinx query syslogged. Then re-run the >>> pilerexport command, and record the new sphinx query, and compare it >>> with the previous value. >>> >>> Verify that even the single-quotes and double quotes are the same in >>> both queries. >>> >>> Janos SUTO >>> >>> >>> On 2021-04-07 18:18, Ryan Blenis wrote: >>> > Hi Janos, >>> > >>> > I have to export potentially a ton of emails and was looking to use >>> > pilerexport versus multiple batches of GUI searches. I saw the -w flag >>> > and thought "great, I can use this" but it doesn't seem to respond >>> > appropriately for my test case. I have 2 emails that match the >>> > following (generalized terms used vs actual), limiting with -m 3 for >>> > testing purposes (I should only get 2 back). >>> > >>> > pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w >>> > 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search >>> > term)|termin*)'"'"')' >>> > >>> > Now, that match is just the bash string escaped version of: >>> > MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search >>> > term)|termin*)') >>> > (That's j
Re: pilerexport w flag matching
Disregard that last email. Coffee is good, not re-running ./configure after installing deps is bad. Following up shortly with more pertinent info. Thank you. On Wed, Apr 7, 2021 at 3:58 PM Ryan Blenis wrote: > Hi Janos, > > Thanks for the response, in trying to do this (I cloned the repo, > ./configure --localstatedir=/var --with-database=mariadb , and ran make) > and got this: > > Making all in src > make[1]: Entering directory '/tmp/piler/src/piler/src' > gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2 > -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb > -I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o > pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre > -L/usr/lib/x86_64-linux-gnu/ -lmariadb -L. > /usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file': > /tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to > `zip_open' > /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined > reference to `zip_source_file' > /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined > reference to `zip_file_add' > /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined > reference to `zip_close' > /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined > reference to `zip_strerror' > collect2: error: ld returned 1 exit status > make[1]: *** [Makefile:63: pilerexport] Error 1 > make[1]: Leaving directory '/tmp/piler/src/piler/src' > make: *** [Makefile:41: all-recursive] Error 1 > > (Note that I originally got a zip.h not found error, which I ran apt > install libzip-dev. Ubuntu 20.04.2 LTS > > I can't seem to get past this point to recompile. > > > On Wed, Apr 7, 2021 at 2:50 PM wrote: > >> >> Hello Ryan, >> >> please apply this patch to pilerexport.c, and recompile it. >> >> https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d >> >> It syslogs the sphinx query. Then try the following. Enter the search >> query >> on the gui, and record the sphinx query syslogged. Then re-run the >> pilerexport command, and record the new sphinx query, and compare it >> with the previous value. >> >> Verify that even the single-quotes and double quotes are the same in >> both queries. >> >> Janos SUTO >> >> >> On 2021-04-07 18:18, Ryan Blenis wrote: >> > Hi Janos, >> > >> > I have to export potentially a ton of emails and was looking to use >> > pilerexport versus multiple batches of GUI searches. I saw the -w flag >> > and thought "great, I can use this" but it doesn't seem to respond >> > appropriately for my test case. I have 2 emails that match the >> > following (generalized terms used vs actual), limiting with -m 3 for >> > testing purposes (I should only get 2 back). >> > >> > pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w >> > 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search >> > term)|termin*)'"'"')' >> > >> > Now, that match is just the bash string escaped version of: >> > MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search >> > term)|termin*)') >> > (That's just a fancy sphinx query for "searchterm" within 25 words of >> > MNF OR "search term" OR "term" OR "test search term" or "termin*" for >> > those unfamiliar with sphinx.) >> > >> > Which, when overloading the Advanced Search for the "body" field in >> > the GUI with: >> > searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*) >> > >> > Seems to work just fine and as expected, however, in pilerexport with >> > the aforementioned command I get tons of unrelated emails (not even >> > scoped to the appropriate j...@domain.com recipient). Is using a MATCH >> > term like this with -w possible, or am I looking to do too much here? >> > >> > Note that I saw you added the -o parameter in the source so I may be a >> > version or 2 back (utility doesn't seem to have a -v or --version >> > output), and my version doesn't appear to have that, so I don't really >> > have any great diagnostic/output information to go off of other than >> > the above description. >> > >> > Thank you in advance as always for any insight you can give! >> >
Re: pilerexport w flag matching
Hi Janos, Thanks for the response, in trying to do this (I cloned the repo, ./configure --localstatedir=/var --with-database=mariadb , and ran make) and got this: Making all in src make[1]: Entering directory '/tmp/piler/src/piler/src' gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2 -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb -I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre -L/usr/lib/x86_64-linux-gnu/ -lmariadb -L. /usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file': /tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to `zip_open' /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined reference to `zip_source_file' /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined reference to `zip_file_add' /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined reference to `zip_close' /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined reference to `zip_strerror' collect2: error: ld returned 1 exit status make[1]: *** [Makefile:63: pilerexport] Error 1 make[1]: Leaving directory '/tmp/piler/src/piler/src' make: *** [Makefile:41: all-recursive] Error 1 (Note that I originally got a zip.h not found error, which I ran apt install libzip-dev. Ubuntu 20.04.2 LTS I can't seem to get past this point to recompile. On Wed, Apr 7, 2021 at 2:50 PM wrote: > > Hello Ryan, > > please apply this patch to pilerexport.c, and recompile it. > > https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d > > It syslogs the sphinx query. Then try the following. Enter the search > query > on the gui, and record the sphinx query syslogged. Then re-run the > pilerexport command, and record the new sphinx query, and compare it > with the previous value. > > Verify that even the single-quotes and double quotes are the same in > both queries. > > Janos SUTO > > > On 2021-04-07 18:18, Ryan Blenis wrote: > > Hi Janos, > > > > I have to export potentially a ton of emails and was looking to use > > pilerexport versus multiple batches of GUI searches. I saw the -w flag > > and thought "great, I can use this" but it doesn't seem to respond > > appropriately for my test case. I have 2 emails that match the > > following (generalized terms used vs actual), limiting with -m 3 for > > testing purposes (I should only get 2 back). > > > > pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w > > 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search > > term)|termin*)'"'"')' > > > > Now, that match is just the bash string escaped version of: > > MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search > > term)|termin*)') > > (That's just a fancy sphinx query for "searchterm" within 25 words of > > MNF OR "search term" OR "term" OR "test search term" or "termin*" for > > those unfamiliar with sphinx.) > > > > Which, when overloading the Advanced Search for the "body" field in > > the GUI with: > > searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*) > > > > Seems to work just fine and as expected, however, in pilerexport with > > the aforementioned command I get tons of unrelated emails (not even > > scoped to the appropriate j...@domain.com recipient). Is using a MATCH > > term like this with -w possible, or am I looking to do too much here? > > > > Note that I saw you added the -o parameter in the source so I may be a > > version or 2 back (utility doesn't seem to have a -v or --version > > output), and my version doesn't appear to have that, so I don't really > > have any great diagnostic/output information to go off of other than > > the above description. > > > > Thank you in advance as always for any insight you can give! >
Re: pilerexport w flag matching
Hello Ryan, please apply this patch to pilerexport.c, and recompile it. https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d It syslogs the sphinx query. Then try the following. Enter the search query on the gui, and record the sphinx query syslogged. Then re-run the pilerexport command, and record the new sphinx query, and compare it with the previous value. Verify that even the single-quotes and double quotes are the same in both queries. Janos SUTO On 2021-04-07 18:18, Ryan Blenis wrote: Hi Janos, I have to export potentially a ton of emails and was looking to use pilerexport versus multiple batches of GUI searches. I saw the -w flag and thought "great, I can use this" but it doesn't seem to respond appropriately for my test case. I have 2 emails that match the following (generalized terms used vs actual), limiting with -m 3 for testing purposes (I should only get 2 back). pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)'"'"')' Now, that match is just the bash string escaped version of: MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)') (That's just a fancy sphinx query for "searchterm" within 25 words of MNF OR "search term" OR "term" OR "test search term" or "termin*" for those unfamiliar with sphinx.) Which, when overloading the Advanced Search for the "body" field in the GUI with: searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*) Seems to work just fine and as expected, however, in pilerexport with the aforementioned command I get tons of unrelated emails (not even scoped to the appropriate j...@domain.com recipient). Is using a MATCH term like this with -w possible, or am I looking to do too much here? Note that I saw you added the -o parameter in the source so I may be a version or 2 back (utility doesn't seem to have a -v or --version output), and my version doesn't appear to have that, so I don't really have any great diagnostic/output information to go off of other than the above description. Thank you in advance as always for any insight you can give!
pilerexport w flag matching
Hi Janos, I have to export potentially a ton of emails and was looking to use pilerexport versus multiple batches of GUI searches. I saw the -w flag and thought "great, I can use this" but it doesn't seem to respond appropriately for my test case. I have 2 emails that match the following (generalized terms used vs actual), limiting with -m 3 for testing purposes (I should only get 2 back). pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)'"'"')' Now, that match is just the bash string escaped version of: MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)') (That's just a fancy sphinx query for "searchterm" within 25 words of MNF OR "search term" OR "term" OR "test search term" or "termin*" for those unfamiliar with sphinx.) Which, when overloading the Advanced Search for the "body" field in the GUI with: searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*) Seems to work just fine and as expected, however, in pilerexport with the aforementioned command I get tons of unrelated emails (not even scoped to the appropriate j...@domain.com recipient). Is using a MATCH term like this with -w possible, or am I looking to do too much here? Note that I saw you added the -o parameter in the source so I may be a version or 2 back (utility doesn't seem to have a -v or --version output), and my version doesn't appear to have that, so I don't really have any great diagnostic/output information to go off of other than the above description. Thank you in advance as always for any insight you can give!