pilerexport where_condition query length limitation

2021-05-03 Thread Ryan Blenis
Hi Janos,

Running some queries with pilerexport -w and it seems I'm running up
against a wall with longer queries (the newly logged "pilerexport" sphinx
query in the mail.log cuts off the query at about ~510 characters) which
causes the query to fail and the following SQL just starts spewing
mysql_stmt errors because the SQL is "incomplete". Assuming that's actually
512 characters with a null termination and another I can't determine just
yet, but looking to hopefully bump up this limit and thought it would also
be a quick bug fix on your end (and make it so I don't have to dig through
too much code to see what needs to be done to make that happen =D )

Thank you!


Re: pilerexport where_condition query length limitation

2021-05-03 Thread Ryan Blenis
Follow up for anyone needing this in the future:

It looks like you can safely change
~line 159 of pilerexport.c (in the run_query function) from:
char s[SMALLBUFSIZE];
to
char s[MAXBUFSIZE];
to get a longer SQL query with pilerexport (from 512 to 8192 characters)

But there should still probably be a more prominent error on the existing
mechanism when the query length exceeds the buffer length, as it currently
only fails in mail.log, and pilerexport itself appears to have just come
back with 0 matches.


On Mon, May 3, 2021 at 6:13 PM Ryan Blenis  wrote:

> Hi Janos,
>
> Running some queries with pilerexport -w and it seems I'm running up
> against a wall with longer queries (the newly logged "pilerexport" sphinx
> query in the mail.log cuts off the query at about ~510 characters) which
> causes the query to fail and the following SQL just starts spewing
> mysql_stmt errors because the SQL is "incomplete". Assuming that's actually
> 512 characters with a null termination and another I can't determine just
> yet, but looking to hopefully bump up this limit and thought it would also
> be a quick bug fix on your end (and make it so I don't have to dig through
> too much code to see what needs to be done to make that happen =D )
>
> Thank you!
>


Re: pilerexport where_condition query length limitation

2021-05-03 Thread sj




Hello Ryan,

I've updated pilerexport.c to have the increased buffer size,
also print the sphinx query to see what's going on with this
commit: 
https://bitbucket.org/jsuto/piler/commits/66a8f60d9d1ced70fe46d6e76041989e7b779f0f


Janos


On 2021-05-04 02:39, Ryan Blenis wrote:

Follow up for anyone needing this in the future:

It looks like you can safely change
~line 159 of pilerexport.c (in the run_query function) from:
char s[SMALLBUFSIZE];
to
char s[MAXBUFSIZE];
to get a longer SQL query with pilerexport (from 512 to 8192
characters)

But there should still probably be a more prominent error on the
existing mechanism when the query length exceeds the buffer length, as
it currently only fails in mail.log, and pilerexport itself appears to
have just come back with 0 matches.

On Mon, May 3, 2021 at 6:13 PM Ryan Blenis 
wrote:


Hi Janos,

Running some queries with pilerexport -w and it seems I'm running up
against a wall with longer queries (the newly logged "pilerexport"
sphinx query in the mail.log cuts off the query at about ~510
characters) which causes the query to fail and the following SQL
just starts spewing mysql_stmt errors because the SQL is
"incomplete". Assuming that's actually 512 characters with a null
termination and another I can't determine just yet, but looking to
hopefully bump up this limit and thought it would also be a quick
bug fix on your end (and make it so I don't have to dig through too
much code to see what needs to be done to make that happen =D )

Thank you!