Re: format string fetch method?

[Willy Tarreau]

On Sat, Oct 08, 2016 at 10:24:35AM +0200, Thierry Fournier wrote:
> Sure, we already speak about this, and if i'm remember correctly, you
> are some difficulties for defining the stack and its consumed memory.
> We can speak about this.

OK great.

> The RPN stack for expression is a great feature. I'm a little sad
> because the Lua will be useless :).

No, Lua won't be useless, it will instead offer a little bit of
overlapping between the two, making people use Lua to write structured
programs and only RPN/fmt strings when having to manipulate formats.
For me it's the exact same difference as between sed and awk.

> In other way, I think that the special sample fetch called "fmt" which
> is taking a format string as parameter and have the sample-fetch
> behaviour is a good idea.
> 
> I think to some attention point like reentrant syntax:
> 
>fmt(%[fmt("%C:%d:%[fmt(%[src])]),crc32()])

Good catch, maybe it was one of the issues as well. To be honnest I don't
remember precisely. But I seem to remember that the parsing context was
limited to one log format per config line for historical reasons and that
we needed to break and redesign the whole parser chain (4-5 levels of
indirections approx).

Willy

Re: format string fetch method?

[Thierry Fournier]

Hi Willy & Patrick,

On Sat, 8 Oct 2016 10:13:53 +0200
Willy Tarreau  wrote:

> Hi Patrick,
> 
> On Thu, Oct 06, 2016 at 01:37:35PM -0400, Patrick Hemmer wrote:
> > While working with the `http-request set-var` (and a few other places,
> > but primarily here), it would be very useful to be able to use haproxy
> > format strings to define the variable.
> > 
> > For example
> >   http-request set-var(txn.foo) fmt(%ci:%cp:%Ts)
> > Or even
> >   http-request set-var(txn.foo) fmt(%ci:%cp:%Ts),crc32()
> > 
> > I don't currently see a way to do this, but I could be missing something.
> > If it's not possible, any chance of getting it added?
> 
> I agree we're missing this. I've been willing to concatenate certain
> fields without being able to do so. The only way to do it right now is
> to pass via headers which is a pain. I remember contemplating this option
> a few months ago and finding some technical difficulties with it, related
> to the error processing during the parsing but I don't remember exactly
> which ones unfortunately. Probably that should be studied again.
> 
> We also wanted to work on some converters dealing with a stack allowing
> an RPN-like syntax for complex data processing. It would remain compatible
> with what we currently have but offer wider possibilities and cover all
> use cases supported by the fmt string in addition to some cases where the
> fmt does not work (eg: binary lookups in tables). We'd then have some
> strcat() or whatever converters to append the last two strings for example.
> I believe that this option was also what managed to put the fmt string much
> lower in the priority queue.
> 
> Unfortunately everyone willing to work on this has been lacking time and
> now we have neither of them :-(
> 
> Thierry, are you still interested in working on the RPN system and more
> importantly do you think you'll have enough time before the end of this
> month ?


Sure, we already speak about this, and if i'm remember correctly, you
are some difficulties for defining the stack and its consumed memory.
We can speak about this.

The RPN stack for expression is a great feature. I'm a little sad
because the Lua will be useless :).

In other way, I think that the special sample fetch called "fmt" which
is taking a format string as parameter and have the sample-fetch
behaviour is a good idea.

I think to some attention point like reentrant syntax:

   fmt(%[fmt("%C:%d:%[fmt(%[src])]),crc32()])

Thierry


> Thanks,
> Willy


-- 
Thierry Fournier
Web Performance & Security Expert
m: +33 6 68 69 21 85  | e: thierry.fourn...@ozon.io
w: http://www.ozon.io/| b: http://blog.ozon.io/

format string fetch method?

[Patrick Hemmer]

While working with the `http-request set-var` (and a few other places,
but primarily here), it would be very useful to be able to use haproxy
format strings to define the variable.

For example
  http-request set-var(txn.foo) fmt(%ci:%cp:%Ts)
Or even
  http-request set-var(txn.foo) fmt(%ci:%cp:%Ts),crc32()

I don't currently see a way to do this, but I could be missing something.
If it's not possible, any chance of getting it added?

-Patrick