Re: pilerexport w flag matching
Hello Ryan,
try getting the master branch, do a complete build, however, don't
install it.
Instead enter the src directory, use LD_LIBRARY_PATH=. to make sure
./pilerexport
will use the recently built libpiler.so file, and try again. Also it's
worth to check what's going on 127.0.0.1:9306 with ngrep or similar
tool.
Janos
On 2021-04-08 08:45, Ryan Blenis wrote:
Hi Janos,
Thanks as always for the reply. No worries on the -w replacing the
other switches, a documentation update will clear that up pretty
easily, and thank you, I'd love a total in the dry run!
As for "the export utility assumes searchd is listening on
127.0.0.1:9306 [1] That part didn't change from 1.3.8 to 1.3.11."
I looked back at all the commits and didn't see anything in that file
that could be causing it, but I'll investigate because now it's just
annoying me that it doesn't work. I saw that it was assuming searchd
was on 127.0.0.1:9306 [1], and it is on my system, however it still
fails with that error in the new build. I don't know if you have a
test case for pilerexport -w, but it should be repeatable. I don't see
much changing in the cfg.c either, but when I printf the cfg.mysqluser
and cfg.mysqlpwd on the old (1.3.8) version I get the correct info. If
I printf cfg.mysqluser and cfg.mysqlpwd on the new version I get "r"
and "" respectively. A quick strace of pilerexport shows it opening
the correct conf file (/usr/local/etc/piler/piler.conf) on both
versions. And specifying the config file with -c did not help either.
Not sure if you can reproduce this on your end or not.
The odd part is, cfg.mysqluser and cfg.mysqlpwd _IS_ utilized in the
"init_session_data(, );" line above the
"init_session_data(, );" and it passes the database "open"
test just fine there...
Thank you.
On Thu, Apr 8, 2021 at 12:13 AM wrote:
Hello Ryan,
On 2021-04-08 01:36, Ryan Blenis wrote:
Thanks, that led me to what is causing the issue / confusion.
The -w switch is described as "Where condition to pass to sphinx,
eg.
"match('@subject: piler')"
Which led me to believe the MATCH string was all that was supposed
to
be there/replaced, however a quick look at the code shows that if
-w
is used, it REPLACES the ENTIRE where clause. This distinction
means
yes, perhaps the "eg." was not that prominent in the short --help
output.
I'll improve the docs on the website, it's lagging behind the actual
features,
and I'll add a clarification on it.
The simplest workaround to this for others would be to note that
-w
allows you to build your own query and negates the use of other
parameters. The ideal fix I think would be to still utilize the
other
parameters, but have -w content appended within the MATCH()
portion of
the query.
Such fix would only complicate things because you can define the
whole
query using -w including the time frame, recipients, etc. Again,
I'll
add a clarification to the docs.
Aside from that: I realize I'm behind on piler (1.3.8), and would
like
to update to get the latest pilerexport with zip capabilities, yet
I
see there is no upgrade information on
https://www.mailpiler.org/wiki/current:upgrade . What is the
process
to the latest (1.3.11)?
Well, simply compile the new stuff, and overwrite the binaries and
the
GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11.
However, don't rush with that. The zip export feature has a poor
performance that needs a rework.
I'd also like to add a "--num-only" type flag to pilerexport to
see
the number of matches before exporting (would probably imply
dryrun).
We have something similar. When specifying -d (or --dry-run) it
prints
the matching serial ids, eg.
$ pilerexport -d -w "MATCH(' some query')"
id:318
id:375
id:518
id:656
id:660
id:688
id:733
I'll improve it to add "total:7" as the last line of the output, if
that's OK.
When trying to just compile the latest, I get the error "error:
cannot
connect to 127.0.0.1:9306 [1] [1]" so I'm not sure if that's an
issue
because not all the components are upgraded, or if I had a
different
configure flag/path configured during the original install.
The export utility assumes searchd is listening on 127.0.0.1:9306
[1]
That part didn't change from 1.3.8 to 1.3.11.
Janos
Links:
--
[1] http://127.0.0.1:9306
Re: pilerexport w flag matching
Hi Janos,
Thanks as always for the reply. No worries on the -w replacing the other
switches, a documentation update will clear that up pretty easily, and
thank you, I'd love a total in the dry run!
As for "the export utility assumes searchd is listening on 127.0.0.1:9306
That part didn't change from 1.3.8 to 1.3.11."
I looked back at all the commits and didn't see anything in that file that
could be causing it, but I'll investigate because now it's just annoying me
that it doesn't work. I saw that it was assuming searchd was on
127.0.0.1:9306, and it is on my system, however it still fails with that
error in the new build. I don't know if you have a test case for
pilerexport -w, but it should be repeatable. I don't see much changing in
the cfg.c either, but when I printf the cfg.mysqluser and cfg.mysqlpwd on
the old (1.3.8) version I get the correct info. If I printf cfg.mysqluser
and cfg.mysqlpwd on the new version I get "r" and "" respectively. A quick
strace of pilerexport shows it opening the correct conf file
(/usr/local/etc/piler/piler.conf) on both versions. And specifying the
config file with -c did not help either. Not sure if you can reproduce this
on your end or not.
The odd part is, cfg.mysqluser and cfg.mysqlpwd _IS_ utilized in the
"init_session_data(, );" line above the
"init_session_data(, );" and it passes the database "open" test
just fine there...
Thank you.
On Thu, Apr 8, 2021 at 12:13 AM wrote:
>
>
> Hello Ryan,
>
> On 2021-04-08 01:36, Ryan Blenis wrote:
> >
> > Thanks, that led me to what is causing the issue / confusion.
> >
> > The -w switch is described as "Where condition to pass to sphinx, eg.
> > "match('@subject: piler')"
> >
> > Which led me to believe the MATCH string was all that was supposed to
> > be there/replaced, however a quick look at the code shows that if -w
> > is used, it REPLACES the ENTIRE where clause. This distinction means
>
> yes, perhaps the "eg." was not that prominent in the short --help
> output.
> I'll improve the docs on the website, it's lagging behind the actual
> features,
> and I'll add a clarification on it.
>
> > The simplest workaround to this for others would be to note that -w
> > allows you to build your own query and negates the use of other
> > parameters. The ideal fix I think would be to still utilize the other
> > parameters, but have -w content appended within the MATCH() portion of
> > the query.
>
> Such fix would only complicate things because you can define the whole
> query using -w including the time frame, recipients, etc. Again, I'll
> add a clarification to the docs.
>
> > Aside from that: I realize I'm behind on piler (1.3.8), and would like
> > to update to get the latest pilerexport with zip capabilities, yet I
> > see there is no upgrade information on
> > https://www.mailpiler.org/wiki/current:upgrade . What is the process
> > to the latest (1.3.11)?
>
> Well, simply compile the new stuff, and overwrite the binaries and the
> GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11.
> However, don't rush with that. The zip export feature has a poor
> performance that needs a rework.
>
> > I'd also like to add a "--num-only" type flag to pilerexport to see
> > the number of matches before exporting (would probably imply dryrun).
>
> We have something similar. When specifying -d (or --dry-run) it prints
> the matching serial ids, eg.
>
> $ pilerexport -d -w "MATCH(' some query')"
> id:318
> id:375
> id:518
> id:656
> id:660
> id:688
> id:733
>
> I'll improve it to add "total:7" as the last line of the output, if
> that's OK.
>
> > When trying to just compile the latest, I get the error "error: cannot
> > connect to 127.0.0.1:9306 [1]" so I'm not sure if that's an issue
> > because not all the components are upgraded, or if I had a different
> > configure flag/path configured during the original install.
>
> The export utility assumes searchd is listening on 127.0.0.1:9306
> That part didn't change from 1.3.8 to 1.3.11.
>
> Janos
>
>
Re: pilerexport w flag matching
Hello Ryan,
On 2021-04-08 01:36, Ryan Blenis wrote:
Thanks, that led me to what is causing the issue / confusion.
The -w switch is described as "Where condition to pass to sphinx, eg.
"match('@subject: piler')"
Which led me to believe the MATCH string was all that was supposed to
be there/replaced, however a quick look at the code shows that if -w
is used, it REPLACES the ENTIRE where clause. This distinction means
yes, perhaps the "eg." was not that prominent in the short --help
output.
I'll improve the docs on the website, it's lagging behind the actual
features,
and I'll add a clarification on it.
The simplest workaround to this for others would be to note that -w
allows you to build your own query and negates the use of other
parameters. The ideal fix I think would be to still utilize the other
parameters, but have -w content appended within the MATCH() portion of
the query.
Such fix would only complicate things because you can define the whole
query using -w including the time frame, recipients, etc. Again, I'll
add a clarification to the docs.
Aside from that: I realize I'm behind on piler (1.3.8), and would like
to update to get the latest pilerexport with zip capabilities, yet I
see there is no upgrade information on
https://www.mailpiler.org/wiki/current:upgrade . What is the process
to the latest (1.3.11)?
Well, simply compile the new stuff, and overwrite the binaries and the
GUI files. The database schema hasn't changed from 1.3.8 to 1.3.11.
However, don't rush with that. The zip export feature has a poor
performance that needs a rework.
I'd also like to add a "--num-only" type flag to pilerexport to see
the number of matches before exporting (would probably imply dryrun).
We have something similar. When specifying -d (or --dry-run) it prints
the matching serial ids, eg.
$ pilerexport -d -w "MATCH(' some query')"
id:318
id:375
id:518
id:656
id:660
id:688
id:733
I'll improve it to add "total:7" as the last line of the output, if
that's OK.
When trying to just compile the latest, I get the error "error: cannot
connect to 127.0.0.1:9306 [1]" so I'm not sure if that's an issue
because not all the components are upgraded, or if I had a different
configure flag/path configured during the original install.
The export utility assumes searchd is listening on 127.0.0.1:9306
That part didn't change from 1.3.8 to 1.3.11.
Janos
Re: pilerexport w flag matching
I should clarify the "error: cannot connect to 127.0.0.1:9306" error
message.
This error does not occur at compile time, but only at runtime of the
latest pilerexport, and only when the -w switch is used.
On Wed, Apr 7, 2021 at 7:36 PM Ryan Blenis wrote:
> Hi Janos,
>
> Thanks, that led me to what is causing the issue / confusion.
>
> The -w switch is described as "Where condition to pass to sphinx, eg.
> "match('@subject: piler')"
>
> Which led me to believe the MATCH string was all that was supposed to be
> there/replaced, however a quick look at the code shows that if -w is used,
> it REPLACES the ENTIRE where clause. This distinction means that the use of
> -w negates the use of the a, b, and r switches as those parameters no
> longer go through your query builder. I was getting more results because it
> wasn't limited to a timeframe or to a recipient due to those flags being
> "skipped".
>
> The simplest workaround to this for others would be to note that -w allows
> you to build your own query and negates the use of other parameters. The
> ideal fix I think would be to still utilize the other parameters, but have
> -w content appended within the MATCH() portion of the query.
>
> Aside from that: I realize I'm behind on piler (1.3.8), and would like to
> update to get the latest pilerexport with zip capabilities, yet I see there
> is no upgrade information on
> https://www.mailpiler.org/wiki/current:upgrade . What is the process to
> the latest (1.3.11)?
>
> I'd also like to add a "--num-only" type flag to pilerexport to see the
> number of matches before exporting (would probably imply dryrun). I didn't
> see a way to do something like that already. If that's something that could
> be added, great, if not, I'll try my hand at it and submit a patch.
>
> When trying to just compile the latest, I get the error "error: cannot
> connect to 127.0.0.1:9306" so I'm not sure if that's an issue because not
> all the components are upgraded, or if I had a different configure
> flag/path configured during the original install.
>
>
> On Wed, Apr 7, 2021 at 5:19 PM Ryan Blenis wrote:
>
>> Disregard that last email. Coffee is good, not re-running ./configure
>> after installing deps is bad. Following up shortly with more pertinent
>> info. Thank you.
>>
>> On Wed, Apr 7, 2021 at 3:58 PM Ryan Blenis wrote:
>>
>>> Hi Janos,
>>>
>>> Thanks for the response, in trying to do this (I cloned the repo,
>>> ./configure --localstatedir=/var --with-database=mariadb , and ran make)
>>> and got this:
>>>
>>> Making all in src
>>> make[1]: Entering directory '/tmp/piler/src/piler/src'
>>> gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2
>>> -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb
>>> -I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o
>>> pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre
>>> -L/usr/lib/x86_64-linux-gnu/ -lmariadb -L.
>>> /usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file':
>>> /tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to
>>> `zip_open'
>>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined
>>> reference to `zip_source_file'
>>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined
>>> reference to `zip_file_add'
>>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined
>>> reference to `zip_close'
>>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined
>>> reference to `zip_strerror'
>>> collect2: error: ld returned 1 exit status
>>> make[1]: *** [Makefile:63: pilerexport] Error 1
>>> make[1]: Leaving directory '/tmp/piler/src/piler/src'
>>> make: *** [Makefile:41: all-recursive] Error 1
>>>
>>> (Note that I originally got a zip.h not found error, which I ran apt
>>> install libzip-dev. Ubuntu 20.04.2 LTS
>>>
>>> I can't seem to get past this point to recompile.
>>>
>>>
>>> On Wed, Apr 7, 2021 at 2:50 PM wrote:
>>>
Hello Ryan,
please apply this patch to pilerexport.c, and recompile it.
https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d
It syslogs the sphinx query. Then try the following. Enter the search
query
on the gui, and record the sphinx query syslogged. Then re-run the
pilerexport command, and record the new sphinx query, and compare it
with the previous value.
Verify that even the single-quotes and double quotes are the same in
both queries.
Janos SUTO
On 2021-04-07 18:18, Ryan Blenis wrote:
> Hi Janos,
>
> I have to export potentially a ton of emails and was looking to use
> pilerexport versus multiple batches of GUI searches. I saw the -w flag
> and thought "great, I can use this" but it doesn't seem to respond
> appropriately for my test case. I have 2 emails that match the
> following (generalized terms used vs actual), limiting with -m 3 for
> testing
Re: pilerexport w flag matching
Hi Janos,
Thanks, that led me to what is causing the issue / confusion.
The -w switch is described as "Where condition to pass to sphinx, eg.
"match('@subject: piler')"
Which led me to believe the MATCH string was all that was supposed to be
there/replaced, however a quick look at the code shows that if -w is used,
it REPLACES the ENTIRE where clause. This distinction means that the use of
-w negates the use of the a, b, and r switches as those parameters no
longer go through your query builder. I was getting more results because it
wasn't limited to a timeframe or to a recipient due to those flags being
"skipped".
The simplest workaround to this for others would be to note that -w allows
you to build your own query and negates the use of other parameters. The
ideal fix I think would be to still utilize the other parameters, but have
-w content appended within the MATCH() portion of the query.
Aside from that: I realize I'm behind on piler (1.3.8), and would like to
update to get the latest pilerexport with zip capabilities, yet I see there
is no upgrade information on https://www.mailpiler.org/wiki/current:upgrade
. What is the process to the latest (1.3.11)?
I'd also like to add a "--num-only" type flag to pilerexport to see the
number of matches before exporting (would probably imply dryrun). I didn't
see a way to do something like that already. If that's something that could
be added, great, if not, I'll try my hand at it and submit a patch.
When trying to just compile the latest, I get the error "error: cannot
connect to 127.0.0.1:9306" so I'm not sure if that's an issue because not
all the components are upgraded, or if I had a different configure
flag/path configured during the original install.
On Wed, Apr 7, 2021 at 5:19 PM Ryan Blenis wrote:
> Disregard that last email. Coffee is good, not re-running ./configure
> after installing deps is bad. Following up shortly with more pertinent
> info. Thank you.
>
> On Wed, Apr 7, 2021 at 3:58 PM Ryan Blenis wrote:
>
>> Hi Janos,
>>
>> Thanks for the response, in trying to do this (I cloned the repo,
>> ./configure --localstatedir=/var --with-database=mariadb , and ran make)
>> and got this:
>>
>> Making all in src
>> make[1]: Entering directory '/tmp/piler/src/piler/src'
>> gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2
>> -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb
>> -I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o
>> pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre
>> -L/usr/lib/x86_64-linux-gnu/ -lmariadb -L.
>> /usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file':
>> /tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to
>> `zip_open'
>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined
>> reference to `zip_source_file'
>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined
>> reference to `zip_file_add'
>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined
>> reference to `zip_close'
>> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined
>> reference to `zip_strerror'
>> collect2: error: ld returned 1 exit status
>> make[1]: *** [Makefile:63: pilerexport] Error 1
>> make[1]: Leaving directory '/tmp/piler/src/piler/src'
>> make: *** [Makefile:41: all-recursive] Error 1
>>
>> (Note that I originally got a zip.h not found error, which I ran apt
>> install libzip-dev. Ubuntu 20.04.2 LTS
>>
>> I can't seem to get past this point to recompile.
>>
>>
>> On Wed, Apr 7, 2021 at 2:50 PM wrote:
>>
>>>
>>> Hello Ryan,
>>>
>>> please apply this patch to pilerexport.c, and recompile it.
>>>
>>> https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d
>>>
>>> It syslogs the sphinx query. Then try the following. Enter the search
>>> query
>>> on the gui, and record the sphinx query syslogged. Then re-run the
>>> pilerexport command, and record the new sphinx query, and compare it
>>> with the previous value.
>>>
>>> Verify that even the single-quotes and double quotes are the same in
>>> both queries.
>>>
>>> Janos SUTO
>>>
>>>
>>> On 2021-04-07 18:18, Ryan Blenis wrote:
>>> > Hi Janos,
>>> >
>>> > I have to export potentially a ton of emails and was looking to use
>>> > pilerexport versus multiple batches of GUI searches. I saw the -w flag
>>> > and thought "great, I can use this" but it doesn't seem to respond
>>> > appropriately for my test case. I have 2 emails that match the
>>> > following (generalized terms used vs actual), limiting with -m 3 for
>>> > testing purposes (I should only get 2 back).
>>> >
>>> > pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w
>>> > 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search
>>> > term)|termin*)'"'"')'
>>> >
>>> > Now, that match is just the bash string escaped version of:
>>> > MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search
>>> > term)|termin*)')
>>> > (That's
Re: pilerexport w flag matching
Disregard that last email. Coffee is good, not re-running ./configure after
installing deps is bad. Following up shortly with more pertinent info.
Thank you.
On Wed, Apr 7, 2021 at 3:58 PM Ryan Blenis wrote:
> Hi Janos,
>
> Thanks for the response, in trying to do this (I cloned the repo,
> ./configure --localstatedir=/var --with-database=mariadb , and ran make)
> and got this:
>
> Making all in src
> make[1]: Entering directory '/tmp/piler/src/piler/src'
> gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2
> -Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb
> -I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o
> pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre
> -L/usr/lib/x86_64-linux-gnu/ -lmariadb -L.
> /usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file':
> /tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to
> `zip_open'
> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined
> reference to `zip_source_file'
> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined
> reference to `zip_file_add'
> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined
> reference to `zip_close'
> /usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined
> reference to `zip_strerror'
> collect2: error: ld returned 1 exit status
> make[1]: *** [Makefile:63: pilerexport] Error 1
> make[1]: Leaving directory '/tmp/piler/src/piler/src'
> make: *** [Makefile:41: all-recursive] Error 1
>
> (Note that I originally got a zip.h not found error, which I ran apt
> install libzip-dev. Ubuntu 20.04.2 LTS
>
> I can't seem to get past this point to recompile.
>
>
> On Wed, Apr 7, 2021 at 2:50 PM wrote:
>
>>
>> Hello Ryan,
>>
>> please apply this patch to pilerexport.c, and recompile it.
>>
>> https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d
>>
>> It syslogs the sphinx query. Then try the following. Enter the search
>> query
>> on the gui, and record the sphinx query syslogged. Then re-run the
>> pilerexport command, and record the new sphinx query, and compare it
>> with the previous value.
>>
>> Verify that even the single-quotes and double quotes are the same in
>> both queries.
>>
>> Janos SUTO
>>
>>
>> On 2021-04-07 18:18, Ryan Blenis wrote:
>> > Hi Janos,
>> >
>> > I have to export potentially a ton of emails and was looking to use
>> > pilerexport versus multiple batches of GUI searches. I saw the -w flag
>> > and thought "great, I can use this" but it doesn't seem to respond
>> > appropriately for my test case. I have 2 emails that match the
>> > following (generalized terms used vs actual), limiting with -m 3 for
>> > testing purposes (I should only get 2 back).
>> >
>> > pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w
>> > 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search
>> > term)|termin*)'"'"')'
>> >
>> > Now, that match is just the bash string escaped version of:
>> > MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search
>> > term)|termin*)')
>> > (That's just a fancy sphinx query for "searchterm" within 25 words of
>> > MNF OR "search term" OR "term" OR "test search term" or "termin*" for
>> > those unfamiliar with sphinx.)
>> >
>> > Which, when overloading the Advanced Search for the "body" field in
>> > the GUI with:
>> > searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)
>> >
>> > Seems to work just fine and as expected, however, in pilerexport with
>> > the aforementioned command I get tons of unrelated emails (not even
>> > scoped to the appropriate j...@domain.com recipient). Is using a MATCH
>> > term like this with -w possible, or am I looking to do too much here?
>> >
>> > Note that I saw you added the -o parameter in the source so I may be a
>> > version or 2 back (utility doesn't seem to have a -v or --version
>> > output), and my version doesn't appear to have that, so I don't really
>> > have any great diagnostic/output information to go off of other than
>> > the above description.
>> >
>> > Thank you in advance as always for any insight you can give!
>>
>
Re: pilerexport w flag matching
Hi Janos,
Thanks for the response, in trying to do this (I cloned the repo,
./configure --localstatedir=/var --with-database=mariadb , and ran make)
and got this:
Making all in src
make[1]: Entering directory '/tmp/piler/src/piler/src'
gcc -std=c99 -O2 -fPIC -Wall -Wextra -Wimplicit-fallthrough=2
-Wuninitialized -Wno-format-truncation -g -I. -I.. -I/usr/include/mariadb
-I/usr/include/mariadb/mysql -D_GNU_SOURCE -DHAVE_TRE -DNEED_MYSQL -o
pilerexport pilerexport.c -lpiler -lz -lm -ldl -lcrypto -lssl -ltre
-L/usr/lib/x86_64-linux-gnu/ -lmariadb -L.
/usr/bin/ld: /tmp/ccU39C8h.o: in function `write_to_zip_file':
/tmp/piler/src/piler/src/pilerexport.c:329: undefined reference to
`zip_open'
/usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:335: undefined
reference to `zip_source_file'
/usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:336: undefined
reference to `zip_file_add'
/usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:342: undefined
reference to `zip_close'
/usr/bin/ld: /tmp/piler/src/piler/src/pilerexport.c:339: undefined
reference to `zip_strerror'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:63: pilerexport] Error 1
make[1]: Leaving directory '/tmp/piler/src/piler/src'
make: *** [Makefile:41: all-recursive] Error 1
(Note that I originally got a zip.h not found error, which I ran apt
install libzip-dev. Ubuntu 20.04.2 LTS
I can't seem to get past this point to recompile.
On Wed, Apr 7, 2021 at 2:50 PM wrote:
>
> Hello Ryan,
>
> please apply this patch to pilerexport.c, and recompile it.
>
> https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d
>
> It syslogs the sphinx query. Then try the following. Enter the search
> query
> on the gui, and record the sphinx query syslogged. Then re-run the
> pilerexport command, and record the new sphinx query, and compare it
> with the previous value.
>
> Verify that even the single-quotes and double quotes are the same in
> both queries.
>
> Janos SUTO
>
>
> On 2021-04-07 18:18, Ryan Blenis wrote:
> > Hi Janos,
> >
> > I have to export potentially a ton of emails and was looking to use
> > pilerexport versus multiple batches of GUI searches. I saw the -w flag
> > and thought "great, I can use this" but it doesn't seem to respond
> > appropriately for my test case. I have 2 emails that match the
> > following (generalized terms used vs actual), limiting with -m 3 for
> > testing purposes (I should only get 2 back).
> >
> > pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w
> > 'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search
> > term)|termin*)'"'"')'
> >
> > Now, that match is just the bash string escaped version of:
> > MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search
> > term)|termin*)')
> > (That's just a fancy sphinx query for "searchterm" within 25 words of
> > MNF OR "search term" OR "term" OR "test search term" or "termin*" for
> > those unfamiliar with sphinx.)
> >
> > Which, when overloading the Advanced Search for the "body" field in
> > the GUI with:
> > searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)
> >
> > Seems to work just fine and as expected, however, in pilerexport with
> > the aforementioned command I get tons of unrelated emails (not even
> > scoped to the appropriate j...@domain.com recipient). Is using a MATCH
> > term like this with -w possible, or am I looking to do too much here?
> >
> > Note that I saw you added the -o parameter in the source so I may be a
> > version or 2 back (utility doesn't seem to have a -v or --version
> > output), and my version doesn't appear to have that, so I don't really
> > have any great diagnostic/output information to go off of other than
> > the above description.
> >
> > Thank you in advance as always for any insight you can give!
>
Re: pilerexport w flag matching
Hello Ryan,
please apply this patch to pilerexport.c, and recompile it.
https://bitbucket.org/jsuto/piler/commits/e6607b0bf1d44562bcf2a08e3bfed94181b7b95d
It syslogs the sphinx query. Then try the following. Enter the search
query
on the gui, and record the sphinx query syslogged. Then re-run the
pilerexport command, and record the new sphinx query, and compare it
with the previous value.
Verify that even the single-quotes and double quotes are the same in
both queries.
Janos SUTO
On 2021-04-07 18:18, Ryan Blenis wrote:
Hi Janos,
I have to export potentially a ton of emails and was looking to use
pilerexport versus multiple batches of GUI searches. I saw the -w flag
and thought "great, I can use this" but it doesn't seem to respond
appropriately for my test case. I have 2 emails that match the
following (generalized terms used vs actual), limiting with -m 3 for
testing purposes (I should only get 2 back).
pilerexport -a 2010.10.01 -b 2021.04.06 -r "j...@domain.com" -m 3 -w
'MATCH('"'"'searchterm NEAR/25 (MNF|(search term)|term|(test search
term)|termin*)'"'"')'
Now, that match is just the bash string escaped version of:
MATCH('searchterm NEAR/25 (MNF|(search term)|term|(test search
term)|termin*)')
(That's just a fancy sphinx query for "searchterm" within 25 words of
MNF OR "search term" OR "term" OR "test search term" or "termin*" for
those unfamiliar with sphinx.)
Which, when overloading the Advanced Search for the "body" field in
the GUI with:
searchterm NEAR/25 (MNF|(search term)|term|(test search term)|termin*)
Seems to work just fine and as expected, however, in pilerexport with
the aforementioned command I get tons of unrelated emails (not even
scoped to the appropriate j...@domain.com recipient). Is using a MATCH
term like this with -w possible, or am I looking to do too much here?
Note that I saw you added the -o parameter in the source so I may be a
version or 2 back (utility doesn't seem to have a -v or --version
output), and my version doesn't appear to have that, so I don't really
have any great diagnostic/output information to go off of other than
the above description.
Thank you in advance as always for any insight you can give!
