Re: cgit-lua: to jit or not to jit

[Andrew Starks]

On Monday, January 13, 2014, Jason A. Donenfeld wrote:

> On Tue, Jan 14, 2014 at 3:37 AM, Andrew Starks 
> >
> wrote:
> >
> > I don't have LuaJit installed and would not install it (and migrate
> > everything I do over to it) just to use a library. By way of example,
> your
> > library may as well have been written for Python, for as much good as it
> > would be to me.
> >
> > By contrast, if you stick to the subset of 5.2 that 5.1 supports, and /
> or
> > use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
> > 5.2 can use it.
> >
> > The question, from a user's perspective is: what benefit are you giving
> me,
> > in exchange for locking me into luajit, as a dependency?
> >
> > Even if I am using Luajit, that doesn't mean that I don't need to support
> > the current, mainline distribution and straight 5.1. So, I can't use your
> > library as a dependency, if this were the case.
> >
> > It's easier for you if you like what the FFI gives you. Supporting the
> > common subset and using luacompat, as necessary, is the simplest, for the
> > user.
> >
> > IMHO, of course
>
> That's a fairly compelling opinion. The only thing against it is the
> temptation of using FFI in the default scripts that we ship with cgit.
> But I suppose for the sake of giving users choice later on, it might
> be best, as you've said, to continue to support both, and let the user
> choose.
>

Jason,

I also just remembered this:

https://github.com/jmckaskill/luaffi

Which is a luajit compatible FFI extension for Lua 5.1 and Lua 5.2, but 5.2
is listed as beta. It might be worth a shot, if it lets you gain some of
those conveniences and keep a broad support base.

-Andrew
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: cgit-lua: to jit or not to jit

[Andrew Starks]

On Monday, January 13, 2014, Jason A. Donenfeld wrote:

> On Tue, Jan 14, 2014 at 3:37 AM, Andrew Starks 
> >
> wrote:
> >
> > I don't have LuaJit installed and would not install it (and migrate
> > everything I do over to it) just to use a library. By way of example,
> your
> > library may as well have been written for Python, for as much good as it
> > would be to me.
> >
> > By contrast, if you stick to the subset of 5.2 that 5.1 supports, and /
> or
> > use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
> > 5.2 can use it.
> >
> > The question, from a user's perspective is: what benefit are you giving
> me,
> > in exchange for locking me into luajit, as a dependency?
> >
> > Even if I am using Luajit, that doesn't mean that I don't need to support
> > the current, mainline distribution and straight 5.1. So, I can't use your
> > library as a dependency, if this were the case.
> >
> > It's easier for you if you like what the FFI gives you. Supporting the
> > common subset and using luacompat, as necessary, is the simplest, for the
> > user.
> >
> > IMHO, of course
>
> That's a fairly compelling opinion. The only thing against it is the
> temptation of using FFI in the default scripts that we ship with cgit.


This is a great point, as well. Also, there are some spiciffic use cases
where speed might be critical and luajit has some common cases where it
really shines, no doubt.

But I suppose for the sake of giving users choice later on, it might
> be best, as you've said, to continue to support both, and let the user
> choose.
>

I'll also say that I will make it a point to check this out. I hadn't heard
of the project before and it sounds interesting.

-Andrew
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: cgit-lua: to jit or not to jit

[Jason A. Donenfeld]

On Tue, Jan 14, 2014 at 3:37 AM, Andrew Starks  wrote:
>
> I don't have LuaJit installed and would not install it (and migrate
> everything I do over to it) just to use a library. By way of example, your
> library may as well have been written for Python, for as much good as it
> would be to me.
>
> By contrast, if you stick to the subset of 5.2 that 5.1 supports, and / or
> use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
> 5.2 can use it.
>
> The question, from a user's perspective is: what benefit are you giving me,
> in exchange for locking me into luajit, as a dependency?
>
> Even if I am using Luajit, that doesn't mean that I don't need to support
> the current, mainline distribution and straight 5.1. So, I can't use your
> library as a dependency, if this were the case.
>
> It's easier for you if you like what the FFI gives you. Supporting the
> common subset and using luacompat, as necessary, is the simplest, for the
> user.
>
> IMHO, of course

That's a fairly compelling opinion. The only thing against it is the
temptation of using FFI in the default scripts that we ship with cgit.
But I suppose for the sake of giving users choice later on, it might
be best, as you've said, to continue to support both, and let the user
choose.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: cgit-lua: to jit or not to jit

[Jason A. Donenfeld]

On Tue, Jan 14, 2014 at 3:34 AM, William Ahern
 wrote:
> The 2GB memory limit on 64-bit was a huge blocker at my work. We almost
> immeditely ran afoul of it and have chosen to stabilize on Lua 5.2 for one
> of our cloud services. (On 16-way SMP, 32GB RAM boxes it's more important to
> be able to scale up and out then to have JITd loops, especially when the
> most performance sensitive code is in C anyhow.)
>
> I suspect that if you ended up trying to write a full OpenSSL wrapper using
> LuaJIT's FFI that you wouldn't save very much time and effort. I say that as
> someone having written the most comprehensive OpenSSL bindings in Lua:
>
> http://25thandclement.com/~william/projects/luaossl.html
>
> Things might be different if you just bound one or two interfaces. LuaJITs
> FFI excels at that sort of ad hoc interface binding. But for full-blown
> bindings things get more complicated. OpenSSL, for example has a ton of
> preprocessor generated tags. Also, string and buffer management can be made
> significantly more efficient in C, so when you're aggregating results into a
> large buffer before pushing onto the Lua stack, it's easier to get good
> performance with C code. For larger or complex modules the payoffs with
> LuaJIT FFI diminish. So it all depends on your context. How sophisticated
> will your bindings be?

In this case, it'd just be calculating an md5 of an email address to
display a gravatar. So at the moment, super simple. Good point about
the cost of aggregation for bigger uses, though.

Generally, our use case is -- cgit spits out a bunch of HTML.
Sometimes sysadmins might want to modify it on the fly based on
various pieces of information -- for example, adding a gravatar image
next to mentions of an author based on email address -- in which case,
the admin codes up a simple lua file [1]. Super super simple usage.

>
> Because Lua is emebedded in your project it's not that big of deal to just
> settle on LuaJIT. Having to deal with the different versions is more of a
> problem for a module writer, where you run into problems with differences
> across versions and implementations--e.g. GC (Lua 5.2 has ephemeron tables)
> or internals (e.g. FILE* handles).
>
>

That's a good point, okay.

[1] http://git.zx2c4.com/cgit/tree/filters/email-gravatar.lua
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: cgit-lua: to jit or not to jit

[Andrew Starks]

On Monday, January 13, 2014, Jason A. Donenfeld wrote:

> On Tue, Jan 14, 2014 at 3:12 AM, Tim Hill >
> wrote:
> > LuaJIT is currently compatible with Lua 5.1, not 5.2 or the upcoming
> 5.3, and my understanding is the LuaJIT project has now forked from
> mainstream Lua and will not be staying in sync with anything past 5.1, so I
> would expect over time the two to diverge.
>
> AFAIK, LuaJIT supports 5.2 currently via the
> -DLUAJIT_ENABLE_LUA52COMPA flag and mentions it here
> http://luajit.org/extensions.html#lua52 . OTOH, their roadmap page
> says:
>
> > As I've previously said, Lua 5.2 provides few tangible benefits.
> > LuaJIT already includes the major new features, without breaking
> > compatibility. Upgrading to be compatible with 5.2, just for the
> > sake of a higher version number, is neither a priority nor a
> > sensible move for most LuaJIT users.
>
> So I'm not sure what to think.
>
>
I don't have LuaJit installed and would not install it (and migrate
everything I do over to it) just to use a library. By way of example, your
library may as well have been written for Python, for as much good as it
would be to me.

By contrast, if you stick to the subset of 5.2 that 5.1 supports, and / or
use a bit of the luacomp library, then anyone with lua 5.1, luajit or Lua
5.2 can use it.

The question, from a user's perspective is: what benefit are you giving me,
in exchange for locking me into luajit, as a dependency?

Even if I am using Luajit, that doesn't mean that I don't need to support
the current, mainline distribution and straight 5.1. So, I can't use your
library as a dependency, if this were the case.

It's easier for you if you like what the FFI gives you. Supporting the
common subset and using luacompat, as necessary, is the simplest, for the
user.

IMHO, of course

-Andrew
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: cgit-lua: to jit or not to jit

[Jason A. Donenfeld]

On Tue, Jan 14, 2014 at 3:12 AM, Tim Hill  wrote:
> LuaJIT is currently compatible with Lua 5.1, not 5.2 or the upcoming 5.3, and 
> my understanding is the LuaJIT project has now forked from mainstream Lua and 
> will not be staying in sync with anything past 5.1, so I would expect over 
> time the two to diverge.

AFAIK, LuaJIT supports 5.2 currently via the
-DLUAJIT_ENABLE_LUA52COMPA flag and mentions it here
http://luajit.org/extensions.html#lua52 . OTOH, their roadmap page
says:

> As I've previously said, Lua 5.2 provides few tangible benefits.
> LuaJIT already includes the major new features, without breaking
> compatibility. Upgrading to be compatible with 5.2, just for the
> sake of a higher version number, is neither a priority nor a
> sensible move for most LuaJIT users.

So I'm not sure what to think.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: cgit-lua: to jit or not to jit

[Jason A. Donenfeld]

On Tue, Jan 14, 2014 at 3:10 AM, demetri  wrote:
> 1) PUC Lua works on more machines/architectures (though Mike has
> substantially closed the gap for most architectures people care about in a
> non-embedded non-mobile environment)

Do you know what the gap is?

LuaJIT supports x86, x86_x64, arm, ppc, e500, and mips. I guess that
leaves, what? Sparc, AVR, ia64, ... other obscure ones? Hmm..

>
> 2) some Linux distributions include PUC Lua but few have LuaJIT installed
> by default, so from the user's perspective LuaJIT is one more dependency.
> Some people care about this and others don't.

Good point.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH v3 2/2] filters: Choose 'trac' theme in pygments

[Jason A. Donenfeld]

Personally, I think the trac colors are a bit ugly. I like pastie
best. But this is just preference.

Here's a comparison site:
http://blog.favrik.com/2011/02/22/preview-all-pygments-styles-for-your-code-highlighting-needs/

If folks want to take some kind of vote, I'll go with majority opinion.

pastie: 1
trac: 1
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH] cgitrc.5.txt: Fix documentation of the snapshot mask

[Jason A. Donenfeld]

Merged, thanks.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

cgit-lua: to jit or not to jit

[Jason A. Donenfeld]

Hi folks,

Over at the cgit project [1], we're in the process of adding Lua
support. Things are going swimmingly, and I've just merged to the
master tree some commits adding full lua support [2] to our filter API
[3] and an example script [4]. At the moment we've got some moderately
messy logic in our makefile [6] that autodetects if LuaJIT is
installed, and if it isn't, falls back to mainline Lua.

What I'm wondering is -- what is the purpose of keeping around support
for mainline Lua? Why shouldn't we just go with LuaJIT and be done
with it? I'm unable to find any advantages mainline Lua has over
LuaJIT, which is why I'm emailing these lists. What features am I
forgetting about that would make some people prefer mainline Lua over
LuaJIT?

An additional motivation for wanting to go to LuaJIT is the FFI
library that comes with it [5]. It seems like this would make
packaging and swapping scripts a lot easier. For example, in the
gravatar script [4], I depend on luacrypto, which some users might not
want to install or have easily available. With FFI, I could instead
just wrap OpenSSL (which cgit already depends on). But perhaps there
are some obvious downsides to this approach that I also am missing.

My experience with embedding Lua inside cgit so far has been very
nice. What a clean simple API. I look forward to hearing your
responses and any feedback and suggestions you might have.

Thank you,
Jason Donenfeld


[1] http://git.zx2c4.com/cgit/about/
[2] http://git.zx2c4.com/cgit/tree/filter.c#n176
[3] http://git.zx2c4.com/cgit/tree/cgitrc.5.txt#n569
[4] http://git.zx2c4.com/cgit/tree/filters/email-gravatar.lua
[5] http://luajit.org/ext_ffi.html
[6] http://git.zx2c4.com/cgit/tree/cgit.mk#n30
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: lua vs luajit vs both

[Jason A. Donenfeld]

I've gone ahead and merged the lua work to master, for testing and
subsequent cleanup before release.

Regarding "to jit or not to jit", I currently have this fancy autodetection
logic:
http://git.zx2c4.com/cgit/commit/?id=3488d124052f5c3ddef303ed5306ad6a458794c1

John -- I'm waiting for your input on the parent email, as you seem to be
the originator of the opinion that "both are good".

Please -- everybody -- test master and let me know any regressions.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH] cgitrc.5.txt: Fix documentation of the snapshot mask

[Lukas Fleischer]

Mention that the snapshot setting only specifies the formats that links
are generated for and not the set of formats that are accessible via
HTTP.

Signed-off-by: Lukas Fleischer 
---
 cgitrc.5.txt | 13 +++--
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/cgitrc.5.txt b/cgitrc.5.txt
index 52caed0..7e9eaeb 100644
--- a/cgitrc.5.txt
+++ b/cgitrc.5.txt
@@ -382,10 +382,10 @@ side-by-side-diffs::
default. Default value: "0".
 
 snapshots::
-   Text which specifies the default set of snapshot formats generated by
-   cgit. The value is a space-separated list of zero or more of the
-   values "tar", "tar.gz", "tar.bz2", "tar.xz" and "zip". Default value:
-   none.
+   Text which specifies the default set of snapshot formats that cgit
+   generates links for. The value is a space-separated list of zero or
+   more of the values "tar", "tar.gz", "tar.bz2", "tar.xz" and "zip".
+   Default value: none.
 
 source-filter::
Specifies a command which will be invoked to format plaintext blobs
@@ -526,8 +526,9 @@ repo.readme::
file. Default value: .
 
 repo.snapshots::
-   A mask of allowed snapshot-formats for this repo, restricted by the
-   "snapshots" global setting. Default value: .
+   A mask of snapshot formats for this repo that cgit generates links for,
+   restricted by the global "snapshots" setting. Default value:
+   .
 
 repo.section::
Override the current section name for this repository. Default value:
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH v3 1/2] filters: Improved syntax-highlighting.py

[Stefan Tatschner]

Am 13.01.2014 23:16, schrieb Jason A. Donenfeld:
> On Mon, Jan 13, 2014 at 11:13 PM, Stefan Tatschner  
> wrote:
>> Why did you apply my patch with tabs instead of spaces? I was wondering
>> because I adjusted the python script according to pep8 [1] and I'm sure
>> the patchfile was with spaces. Maybe you have an automatic convert script?
> 
> I like tabs. They're for tabbing. There's a character for doing the
> thing that they do. I like to use that character. The rest of cgit is
> the same way. So, python scripts in the tree follow suit.

got it. :)




signature.asc
Description: OpenPGP digital signature
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH v3 1/2] filters: Improved syntax-highlighting.py

[Jason A. Donenfeld]

On Mon, Jan 13, 2014 at 11:13 PM, Stefan Tatschner  wrote:
> Why did you apply my patch with tabs instead of spaces? I was wondering
> because I adjusted the python script according to pep8 [1] and I'm sure
> the patchfile was with spaces. Maybe you have an automatic convert script?

I like tabs. They're for tabbing. There's a character for doing the
thing that they do. I like to use that character. The rest of cgit is
the same way. So, python scripts in the tree follow suit.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH v3 1/2] filters: Improved syntax-highlighting.py

[Stefan Tatschner]

Am 13.01.2014 22:50, schrieb Jason A. Donenfeld:
> Perfect! Applied. Thanks for going through all the revisions.

thanks as well. :)
I have one simple question (just out of curiosity; I do not want to bash
anybody).

Why did you apply my patch with tabs instead of spaces? I was wondering
because I adjusted the python script according to pep8 [1] and I'm sure
the patchfile was with spaces. Maybe you have an automatic convert script?

[1] http://www.python.org/dev/peps/pep-0008/#tabs-or-spaces



signature.asc
Description: OpenPGP digital signature
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH v3 1/2] filters: Improved syntax-highlighting.py

[Jason A. Donenfeld]

Perfect! Applied. Thanks for going through all the revisions.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v3 1/2] filters: Improved syntax-highlighting.py

[Stefan Tatschner]

- Switched back to python2 according to a problem in pygments with python3.
  With the next release of pygments this problem should be fixed.
  Issue see here:
  https://bitbucket.org/birkenfeld/pygments-main/issue/901/problems-with-python3
- Just read the stdin, decode it to utf-8 and ignore unknown signs. This ensures
  that even destroyed files do not cause any errors in the filter.
- Improved language guessing:
  -> At first use guess_lexer_for_filename for a better detection of the used
 programming languages (even mixed cases will be detected, e.g. php + html).
  -> If nothing was found look if there is a shebang and use guess_lexer.
  -> As default/fallback choose TextLexer.

Signed-off-by: Stefan Tatschner 
---
 filters/syntax-highlighting.py | 52 +++---
 1 file changed, 33 insertions(+), 19 deletions(-)

diff --git a/filters/syntax-highlighting.py b/filters/syntax-highlighting.py
index 72d9097..53081a4 100755
--- a/filters/syntax-highlighting.py
+++ b/filters/syntax-highlighting.py
@@ -1,13 +1,16 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python2
 
-# This script uses Pygments and Python3. You must have both installed for this 
to work.
+# This script uses Pygments and Python2. You must have both installed
+# for this to work.
+#
 # http://pygments.org/
 # http://python.org/
 #
-# It may be used with the source-filter or repo.source-filter settings in 
cgitrc.
+# It may be used with the source-filter or repo.source-filter settings
+# in cgitrc.
 #
-# The following environment variables can be used to retrieve the configuration
-# of the repository for which this script is called:
+# The following environment variables can be used to retrieve the
+# configuration of the repository for which this script is called:
 # CGIT_REPO_URL( = repo.url   setting )
 # CGIT_REPO_NAME   ( = repo.name  setting )
 # CGIT_REPO_PATH   ( = repo.path  setting )
@@ -18,22 +21,33 @@
 
 
 import sys
-import cgi
-import codecs
-from pygments.lexers import get_lexer_for_filename
 from pygments import highlight
+from pygments.util import ClassNotFound
+from pygments.lexers import TextLexer
+from pygments.lexers import guess_lexer
+from pygments.lexers import guess_lexer_for_filename
 from pygments.formatters import HtmlFormatter
 
-sys.stdin = codecs.getreader("utf-8")(sys.stdin.detach())
-sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
-doc = sys.stdin.read()
+
+# read stdin and decode to utf-8. ignore any unkown signs.
+data = sys.stdin.read().decode(encoding='utf-8', errors='ignore')
+filename = sys.argv[1]
+formatter = HtmlFormatter(encoding='utf-8', style='pastie')
+
 try:
-   lexer = get_lexer_for_filename(sys.argv[1])
-   formatter = HtmlFormatter(style='pastie')
-   sys.stdout.write("")
-   sys.stdout.write(formatter.get_style_defs('.highlight'))
-   sys.stdout.write("")
+lexer = guess_lexer_for_filename(filename, data, encoding='utf-8')
+except ClassNotFound:
+# check if there is any shebang
+if data[0:2] == '#!':
+lexer = guess_lexer(data, encoding='utf-8')
+else:
+lexer = TextLexer(encoding='utf-8')
+except TypeError:
+lexer = TextLexer(encoding='utf-8')
 
-   highlight(doc, lexer, formatter, sys.stdout)
-except:
-   sys.stdout.write(str(cgi.escape(doc).encode("ascii", 
"xmlcharrefreplace"), "ascii"))
+# highlight! :-)
+# printout pygments' css definitions as well
+sys.stdout.write('')
+sys.stdout.write(formatter.get_style_defs('.highlight'))
+sys.stdout.write('')
+highlight(data, lexer, formatter, outfile=sys.stdout)
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v3 2/2] filters: Choose 'trac' theme in pygments

[Stefan Tatschner]

Using trac theme for pygments. It is very clean and not as
intrusive as the default or pastie theme. Especially I do
not like the the 'pastie' theme very much because of the
very strange illustration of multiline strings (the red
background thing).

Signed-off-by: Stefan Tatschner 
---
 filters/syntax-highlighting.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/filters/syntax-highlighting.py b/filters/syntax-highlighting.py
index 53081a4..67855d1 100755
--- a/filters/syntax-highlighting.py
+++ b/filters/syntax-highlighting.py
@@ -32,7 +32,7 @@ from pygments.formatters import HtmlFormatter
 # read stdin and decode to utf-8. ignore any unkown signs.
 data = sys.stdin.read().decode(encoding='utf-8', errors='ignore')
 filename = sys.argv[1]
-formatter = HtmlFormatter(encoding='utf-8', style='pastie')
+formatter = HtmlFormatter(encoding='utf-8', style='trac')
 
 try:
 lexer = guess_lexer_for_filename(filename, data, encoding='utf-8')
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

lua vs luajit vs both

[Jason A. Donenfeld]

Hi,

What reasons do we have for supporting lua at all? Why not just go
with luajit? It's faster and just as widely supported. The motivation
for not supporting vanilla lua is this luajit library:
http://luajit.org/ext_ffi.html . This would be a nice way of being
able to ship scripts without a big comment in the header "you need to
have the luacrypto package installed for this to work", and similar.
Currently, the tree supports both lua and luajit, but I'm tempted to
chop this down to just luajit. Thoughts on this?

Jason
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH v2] filters: Improved syntax-highlighting.py

[Stefan Tatschner]

Am 13.01.2014 15:50, schrieb Jason A. Donenfeld:
> On Mon, Jan 13, 2014 at 3:47 PM, Stefan Tatschner  
> wrote:
>> No, that's not the issue. By default 'pygments' adds a > style="line-height: 125%"> tag so I had to change the line-height of the
>> other column as well.
> 
> Strange. It doesn't do that on the current script. Is this because of
> the inline styles? In which case, since we're reverting to the 

[PATCH v2] filters: Improved syntax-highlighting.py

[Jason A. Donenfeld]

On Mon, Jan 13, 2014 at 3:47 PM, Stefan Tatschner  wrote:
> No, that's not the issue. By default 'pygments' adds a  style="line-height: 125%"> tag so I had to change the line-height of the
> other column as well.

Strange. It doesn't do that on the current script. Is this because of
the inline styles? In which case, since we're reverting to the 

Re: [PATCH v2] filters: Improved syntax-highlighting.py

[Stefan Tatschner]

Am 13.01.2014 15:21, schrieb Jason A. Donenfeld:
> On Mon, Jan 13, 2014 at 12:02 PM, Stefan Tatschner  
> wrote:
>> - Using inline CSS instead of this sys.stdout.print() hack.
> 
> Please don't do this. Inline CSS makes for much bigger files, and
> there's nothing in the HTML5 spec that forbids us from putting 

Re: [RESEND PATCH 1/1] enable cgit to show gravatar for author, committer and tagger

[Jason A. Donenfeld]

On Mon, Jan 13, 2014 at 3:18 PM, Christian Hesse  wrote:
> Your avatar (and mine as well) are fine when scaled down. But if there is no
> avatar on gravatar you get the "arcade-style pixelated faces", which look
> very fuzzy when scaled by the browser...

Good point. Excuse me while I make a trivial change in the fancy new
lua script... done! It's working! : )

You're right -- the arcade faces do look a lot better at 13px. I'll
set s=13 for v2 of the filter-infra series currently going on.
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH v2] filters: Improved syntax-highlighting.py

[Jason A. Donenfeld]

Thanks for all your hard work on this. Sorry for the extended back and
forth. More comments, alas alas, below.

On Mon, Jan 13, 2014 at 12:02 PM, Stefan Tatschner  wrote:
> - Using inline CSS instead of this sys.stdout.print() hack.

Please don't do this. Inline CSS makes for much bigger files, and
there's nothing in the HTML5 spec that forbids us from putting 

Re: [RESEND PATCH 1/1] enable cgit to show gravatar for author, committer and tagger

[Christian Hesse]

"Jason A. Donenfeld"  on Thu, 2014/01/09 16:19:
> On Thu, Jan 9, 2014 at 10:18 AM, Christian Hesse  wrote:
> > You modified the code to make the icon match the font size, which is 10pt
> > by default. This is hard coded to the css file. Please note the URL
> > includes a size argument. Does it make sense to just retrieve a 10x10
> > pixel image?
> 
> Generally, 10pt is actually 13px, not 10px. I did see the size
> argument of the URL, and played with changing this. But I was thinking
> that often times, I make the font size slightly bigger, and requesting
> the 3px larger image (16px instead of 13px) gives a little bit of
> leeway for this sort of forced scaling. For my particular avatar, the
> difference winds up being 13 bytes of data.

Your avatar (and mine as well) are fine when scaled down. But if there is no
avatar on gravatar you get the "arcade-style pixelated faces", which look
very fuzzy when scaled by the browser...
-- 
main(a){char*c=/*Schoene Gruesse */"B?IJj;MEH"
"CX:;",b;for(a/*Chris   get my mail address:*/=0;b=c[a++];)
putchar(b-1/(/*   gcc -o sig sig.c && ./sig*/b/42*2-3)*42);}


signature.asc
Description: PGP signature
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v2 6/9] filter: add lua support

[Jason A. Donenfeld]

Signed-off-by: Jason A. Donenfeld 
---
 cgit.mk  |  22 ++-
 cgitrc.5.txt |  29 ++
 filter.c | 186 +++
 3 files changed, 234 insertions(+), 3 deletions(-)

diff --git a/cgit.mk b/cgit.mk
index 9d6dea8..25f2eab 100644
--- a/cgit.mk
+++ b/cgit.mk
@@ -25,6 +25,25 @@ ifdef NO_C99_FORMAT
CFLAGS += -DNO_C99_FORMAT
 endif
 
+ifdef NO_LUA
+   CFLAGS += -DNO_LUA
+else
+
+ifeq (VANILLA,$(LUA_IMPLEMENTATION))
+   CFLAGS += -llua
+else
+   LUAJIT_LIBS := $(shell pkg-config --libs luajit)
+   LUAJIT_CFLAGS := $(shell pkg-config --cflags luajit)
+   CGIT_LIBS += $(LUAJIT_LIBS)
+   CFLAGS += $(LUAJIT_CFLAGS)
+endif
+
+endif
+
+CGIT_LIBS += -ldl
+
+
+
 CGIT_OBJ_NAMES += cgit.o
 CGIT_OBJ_NAMES += cache.o
 CGIT_OBJ_NAMES += cmd.o
@@ -61,9 +80,6 @@ $(CGIT_VERSION_OBJS): $(CGIT_PREFIX)VERSION
 $(CGIT_VERSION_OBJS): EXTRA_CPPFLAGS = \
-DCGIT_VERSION='"$(CGIT_VERSION)"'
 
-CGIT_LIBS += -ldl
-
-
 # Git handles dependencies using ":=" so dependencies in CGIT_OBJ are not
 # handled by that and we must handle them ourselves.
 cgit_dep_files := $(foreach f,$(CGIT_OBJS),$(dir $f).depend/$(notdir $f).d)
diff --git a/cgitrc.5.txt b/cgitrc.5.txt
index 60159f6..78f33c8 100644
--- a/cgitrc.5.txt
+++ b/cgitrc.5.txt
@@ -564,6 +564,35 @@ specification with the relevant string; available values 
are:
 'exec:'::
The default "one process per filter" mode.
 
+'lua:'::
+   Executes the script using a built-in Lua interpreter. The script is
+   loaded once per execution of cgit, and may be called multiple times
+   during cgit's lifetime, making it a good choice for repeated filters
+   such as the 'email filter'. It responds to three functions:
+
+   'filter_open(argument1, argument2, argument3, ...)'::
+   This is called upon activation of the filter for a particular
+   set of data.
+   'filter_write(buffer)'::
+   This is called whenever cgit writes data to the webpage.
+   'filter_close()'::
+   This is called when the current filtering operation is
+   completed.
+   
+   Additionally, cgit exposes to the Lua the following built-in functions:
+
+   'html(str)'::
+   Writes 'str' to the webpage.
+   'html_txt(str)'::
+   HTML escapes and writes 'str' to the webpage.
+   'html_attr(str)'::
+   HTML escapes for an attribute and writes "str' to the webpage.
+   'html_url_path(str)'::
+   URL escapes for a path and writes 'str' to the webpage.
+   'html_url_arg(str)'::
+   URL escapes for an argument and writes 'str' to the webpage.
+
+
 Parameters are provided to filters as follows.
 
 about filter::
diff --git a/filter.c b/filter.c
index f5a5992..3702585 100644
--- a/filter.c
+++ b/filter.c
@@ -7,12 +7,19 @@
  */
 
 #include "cgit.h"
+#include "html.h"
 #include 
 #include 
 #include 
 #include 
 #include 
 #include 
+#include 
+#ifndef NO_LUA
+#include 
+#include 
+#include 
+#endif
 
 static ssize_t (*libc_write)(int fd, const void *buf, size_t count);
 static ssize_t (*filter_write)(struct cgit_filter *base, const void *buf, 
size_t count) = NULL;
@@ -164,6 +171,182 @@ void cgit_exec_filter_init(struct cgit_exec_filter 
*filter, char *cmd, char **ar
filter->base.argument_count = 0;
 }
 
+#ifndef NO_LUA
+struct lua_filter {
+   struct cgit_filter base;
+   char *script_file;
+   lua_State *lua_state;
+};
+
+static void error_lua_filter(struct lua_filter *filter)
+{
+   die("Lua error in %s: %s", filter->script_file, 
lua_tostring(filter->lua_state, -1));
+   lua_pop(filter->lua_state, 1);
+}
+
+static ssize_t write_lua_filter(struct cgit_filter *base, const void *buf, 
size_t count)
+{
+   struct lua_filter *filter = (struct lua_filter *) base;
+
+   lua_getglobal(filter->lua_state, "filter_write");
+   lua_pushlstring(filter->lua_state, buf, count);
+   if (lua_pcall(filter->lua_state, 1, 0, 0)) {
+   error_lua_filter(filter);
+   errno = EIO;
+   return -1;
+   }
+   return count;
+}
+
+static inline int hook_lua_filter(lua_State *lua_state, void (*fn)(const char 
*txt))
+{
+   const char *str;
+   ssize_t (*save_filter_write)(struct cgit_filter *base, const void *buf, 
size_t count);
+   struct cgit_filter *save_filter;
+
+   str = lua_tostring(lua_state, 1);
+   if (!str)
+   return 0;
+   
+   save_filter_write = filter_write;
+   save_filter = current_write_filter;
+   unhook_write();
+   fn(str);
+   hook_write(save_filter, save_filter_write);
+
+   return 0;
+}
+
+static int html_lua_filter(lua_State *lua_state)
+{
+   return hook_lua_filter(lua_state, html);
+}
+
+static int html_txt_lua_filter(lua_State *lua_state)
+{
+   return hook_lua_filter(lua_state, html_txt);
+}
+
+static int htm

[PATCH v2 2/9] filter: add interface layer

[Jason A. Donenfeld]

From: John Keeping 

Change the existing cgit_{open,close,fprintf}_filter functions to
delegate to filter-specific implementations accessed via function
pointers on the cgit_filter object.

We treat the "exec" filter type slightly specially here by putting its
structure definition in the header file and providing an "init" function
to set up the function pointers.  This is required so that the
ui-snapshot.c code that applies a compression filter can continue to use
the filter interface to do so.

Signed-off-by: John Keeping 
---
 cgit.h|  8 
 filter.c  | 66 ---
 ui-snapshot.c | 11 +-
 3 files changed, 63 insertions(+), 22 deletions(-)

diff --git a/cgit.h b/cgit.h
index 9b4be26..92e8c55 100644
--- a/cgit.h
+++ b/cgit.h
@@ -57,6 +57,13 @@ typedef enum {
 } filter_type;
 
 struct cgit_filter {
+   int (*open)(struct cgit_filter *, va_list ap);
+   int (*close)(struct cgit_filter *);
+   void (*fprintf)(struct cgit_filter *, FILE *, const char *prefix);
+};
+
+struct cgit_exec_filter {
+   struct cgit_filter base;
char *cmd;
char **argv;
int extra_args;
@@ -346,6 +353,7 @@ extern int cgit_parse_snapshots_mask(const char *str);
 extern int cgit_open_filter(struct cgit_filter *filter, ...);
 extern int cgit_close_filter(struct cgit_filter *filter);
 extern void cgit_fprintf_filter(struct cgit_filter *filter, FILE *f, const 
char *prefix);
+extern void cgit_exec_filter_init(struct cgit_exec_filter *filter, char *cmd, 
char **argv);
 extern struct cgit_filter *cgit_new_filter(const char *cmd, filter_type 
filtertype);
 
 extern void cgit_prepare_repo_env(struct cgit_repo * repo);
diff --git a/filter.c b/filter.c
index 80cf689..0f3edb0 100644
--- a/filter.c
+++ b/filter.c
@@ -13,15 +13,13 @@
 #include 
 #include 
 
-int cgit_open_filter(struct cgit_filter *filter, ...)
+static int open_exec_filter(struct cgit_filter *base, va_list ap)
 {
+   struct cgit_exec_filter *filter = (struct cgit_exec_filter *) base;
int i;
-   va_list ap;
 
-   va_start(ap, filter);
for (i = 0; i < filter->extra_args; i++)
filter->argv[i+1] = va_arg(ap, char *);
-   va_end(ap);
 
filter->old_stdout = chk_positive(dup(STDOUT_FILENO),
"Unable to duplicate STDOUT");
@@ -41,9 +39,9 @@ int cgit_open_filter(struct cgit_filter *filter, ...)
return 0;
 }
 
-
-int cgit_close_filter(struct cgit_filter *filter)
+static int close_exec_filter(struct cgit_filter *base)
 {
+   struct cgit_exec_filter *filter = (struct cgit_exec_filter *) base;
int i, exit_status;
 
chk_non_negative(dup2(filter->old_stdout, STDOUT_FILENO),
@@ -63,21 +61,50 @@ done:
 
 }
 
-void cgit_fprintf_filter(struct cgit_filter *filter, FILE *f, const char 
*prefix)
+static void fprintf_exec_filter(struct cgit_filter *base, FILE *f, const char 
*prefix)
 {
+   struct cgit_exec_filter *filter = (struct cgit_exec_filter *) base;
fprintf(f, "%s%s\n", prefix, filter->cmd);
 }
 
-struct cgit_filter *cgit_new_filter(const char *cmd, filter_type filtertype)
+int cgit_open_filter(struct cgit_filter *filter, ...)
 {
-   struct cgit_filter *f;
-   int args_size = 0;
+   int result;
+   va_list ap;
+   va_start(ap, filter);
+   result = filter->open(filter, ap);
+   va_end(ap);
+   return result;
+}
 
-   if (!cmd || !cmd[0])
-   return NULL;
+int cgit_close_filter(struct cgit_filter *filter)
+{
+   return filter->close(filter);
+}
+
+void cgit_fprintf_filter(struct cgit_filter *filter, FILE *f, const char 
*prefix)
+{
+   filter->fprintf(filter, f, prefix);
+}
+
+void cgit_exec_filter_init(struct cgit_exec_filter *filter, char *cmd, char 
**argv)
+{
+   memset(filter, 0, sizeof(*filter));
+   filter->base.open = open_exec_filter;
+   filter->base.close = close_exec_filter;
+   filter->base.fprintf = fprintf_exec_filter;
+   filter->cmd = cmd;
+   filter->argv = argv;
+}
+
+static struct cgit_filter *new_exec_filter(const char *cmd, filter_type 
filtertype)
+{
+   struct cgit_exec_filter *f;
+   int args_size = 0;
 
-   f = xmalloc(sizeof(struct cgit_filter));
-   memset(f, 0, sizeof(struct cgit_filter));
+   f = xmalloc(sizeof(*f));
+   /* We leave argv for now and assign it below. */
+   cgit_exec_filter_init(f, xstrdup(cmd), NULL);
 
switch (filtertype) {
case SOURCE:
@@ -91,10 +118,17 @@ struct cgit_filter *cgit_new_filter(const char *cmd, 
filter_type filtertype)
break;
}
 
-   f->cmd = xstrdup(cmd);
args_size = (2 + f->extra_args) * sizeof(char *);
f->argv = xmalloc(args_size);
memset(f->argv, 0, args_size);
f->argv[0] = f->cmd;
-   return f;
+   return &f->base;
+}
+
+struct cgit_filter *cgit_new_filter(const char *cmd, filter_type filtertype)
+{
+   

[PATCH v2 8/9] filter: add support for email filter

[Jason A. Donenfeld]

Signed-off-by: Jason A. Donenfeld 
---
 cgit.c   |  6 ++
 cgit.h   |  4 +++-
 cgitrc.5.txt | 18 ++
 filter.c |  3 +++
 shared.c |  1 +
 ui-commit.c  |  4 
 ui-log.c |  2 ++
 ui-refs.c|  9 -
 ui-tag.c |  2 ++
 9 files changed, 47 insertions(+), 2 deletions(-)

diff --git a/cgit.c b/cgit.c
index 725fd65..f3fe56b 100644
--- a/cgit.c
+++ b/cgit.c
@@ -89,6 +89,8 @@ static void repo_config(struct cgit_repo *repo, const char 
*name, const char *va
repo->commit_filter = cgit_new_filter(value, COMMIT);
else if (!strcmp(name, "source-filter"))
repo->source_filter = cgit_new_filter(value, SOURCE);
+   else if (!strcmp(name, "email-filter"))
+   repo->email_filter = cgit_new_filter(value, EMAIL);
}
 }
 
@@ -188,6 +190,8 @@ static void config_cb(const char *name, const char *value)
ctx.cfg.about_filter = cgit_new_filter(value, ABOUT);
else if (!strcmp(name, "commit-filter"))
ctx.cfg.commit_filter = cgit_new_filter(value, COMMIT);
+   else if (!strcmp(name, "email-filter"))
+   ctx.cfg.email_filter = cgit_new_filter(value, EMAIL);
else if (!strcmp(name, "embedded"))
ctx.cfg.embedded = atoi(value);
else if (!strcmp(name, "max-atom-items"))
@@ -711,6 +715,8 @@ static void print_repo(FILE *f, struct cgit_repo *repo)
cgit_fprintf_filter(repo->commit_filter, f, 
"repo.commit-filter=");
if (repo->source_filter && repo->source_filter != ctx.cfg.source_filter)
cgit_fprintf_filter(repo->source_filter, f, 
"repo.source-filter=");
+   if (repo->email_filter && repo->email_filter != ctx.cfg.email_filter)
+   cgit_fprintf_filter(repo->email_filter, f, 
"repo.email-filter=");
if (repo->snapshots != ctx.cfg.snapshots) {
char *tmp = build_snapshot_setting(repo->snapshots);
fprintf(f, "repo.snapshots=%s\n", tmp ? tmp : "");
diff --git a/cgit.h b/cgit.h
index 519d2af..e200a06 100644
--- a/cgit.h
+++ b/cgit.h
@@ -53,7 +53,7 @@ typedef void (*filepair_fn)(struct diff_filepair *pair);
 typedef void (*linediff_fn)(char *line, int len);
 
 typedef enum {
-   ABOUT, COMMIT, SOURCE
+   ABOUT, COMMIT, SOURCE, EMAIL
 } filter_type;
 
 struct cgit_filter {
@@ -99,6 +99,7 @@ struct cgit_repo {
struct cgit_filter *about_filter;
struct cgit_filter *commit_filter;
struct cgit_filter *source_filter;
+   struct cgit_filter *email_filter;
struct string_list submodules;
 };
 
@@ -250,6 +251,7 @@ struct cgit_config {
struct cgit_filter *about_filter;
struct cgit_filter *commit_filter;
struct cgit_filter *source_filter;
+   struct cgit_filter *email_filter;
 };
 
 struct cgit_page {
diff --git a/cgitrc.5.txt b/cgitrc.5.txt
index 78f33c8..b7dc5a4 100644
--- a/cgitrc.5.txt
+++ b/cgitrc.5.txt
@@ -117,6 +117,14 @@ css::
Url which specifies the css document to include in all cgit pages.
Default value: "/cgit.css".
 
+email-filter::
+   Specifies a command which will be invoked to format names and email
+   address of committers, authors, and taggers, as represented in various
+   places throughout the cgit interface. This command will receive an
+   email address as its only command line argument, and the text to
+   format on STDIN. It is to write the formatted text back out onto
+   STDOUT. Default value: none. See also: "FILTER API".
+
 embedded::
Flag which, when set to "1", will make cgit generate a html fragment
suitable for embedding in other html pages. Default value: none. See
@@ -457,6 +465,10 @@ repo.defbranch::
 repo.desc::
The value to show as repository description. Default value: none.
 
+repo.email-filter::
+   Override the default email-filter. Default value: none. See also:
+   "enable-filter-overrides". See also: "FILTER API".
+
 repo.enable-commit-graph::
A flag which can be used to disable the global setting
`enable-commit-graph'. Default value: none.
@@ -607,6 +619,12 @@ commit filter::
be filtered is available on standard input and the filtered text is
expected on standard output.
 
+email filter::
+   This filter is given a single parameter: the email address of the
+   relevent user. The filter will then receive the text string to format
+   on standard input and is expected to write to standard output the
+   formatted text to be included in the page.
+
 source filter::
This filter is given a single parameter: the filename of the source
file to filter. The filter can use the filename to determine (for
diff --git a/filter.c b/filter.c
index 7983737..08ce7a5 100644
--- a/filter.c
+++ b/filter.c
@@ -37,10 +37,12 @@ void cgit_cleanup_filters(void)
reap_filter(ctx.cfg.about_filter);
 

[PATCH v2 7/9] filter: return on null filter from open and close

[Jason A. Donenfeld]

So that we don't have to include the if(filter) open_filter(filter)
block everywhere, we introduce the guard in the function itself. This
should simplify quite a bit of code.

Signed-off-by: Jason A. Donenfeld 
---
 filter.c  |  4 
 ui-commit.c   | 18 ++
 ui-repolist.c |  6 ++
 ui-summary.c  |  8 ++--
 4 files changed, 14 insertions(+), 22 deletions(-)

diff --git a/filter.c b/filter.c
index 3702585..7983737 100644
--- a/filter.c
+++ b/filter.c
@@ -351,6 +351,8 @@ int cgit_open_filter(struct cgit_filter *filter, ...)
 {
int result;
va_list ap;
+   if (!filter)
+   return 0;
va_start(ap, filter);
result = filter->open(filter, ap);
va_end(ap);
@@ -359,6 +361,8 @@ int cgit_open_filter(struct cgit_filter *filter, ...)
 
 int cgit_close_filter(struct cgit_filter *filter)
 {
+   if (!filter)
+   return 0;
return filter->close(filter);
 }
 
diff --git a/ui-commit.c b/ui-commit.c
index aa1892f..5ac79c0 100644
--- a/ui-commit.c
+++ b/ui-commit.c
@@ -107,28 +107,22 @@ void cgit_print_commit(char *hex, const char *prefix)
}
html("\n");
html("");
-   if (ctx.repo->commit_filter)
-   cgit_open_filter(ctx.repo->commit_filter);
+   cgit_open_filter(ctx.repo->commit_filter);
html_txt(info->subject);
-   if (ctx.repo->commit_filter)
-   cgit_close_filter(ctx.repo->commit_filter);
+   cgit_close_filter(ctx.repo->commit_filter);
show_commit_decorations(commit);
html("");
html("");
-   if (ctx.repo->commit_filter)
-   cgit_open_filter(ctx.repo->commit_filter);
+   cgit_open_filter(ctx.repo->commit_filter);
html_txt(info->msg);
-   if (ctx.repo->commit_filter)
-   cgit_close_filter(ctx.repo->commit_filter);
+   cgit_close_filter(ctx.repo->commit_filter);
html("");
if (notes.len != 0) {
html("Notes");
html("");
-   if (ctx.repo->commit_filter)
-   cgit_open_filter(ctx.repo->commit_filter);
+   cgit_open_filter(ctx.repo->commit_filter);
html_txt(notes.buf);
-   if (ctx.repo->commit_filter)
-   cgit_close_filter(ctx.repo->commit_filter);
+   cgit_close_filter(ctx.repo->commit_filter);
html("");
html("");
}
diff --git a/ui-repolist.c b/ui-repolist.c
index 7b1fec3..f9cb21a 100644
--- a/ui-repolist.c
+++ b/ui-repolist.c
@@ -333,9 +333,7 @@ void cgit_print_site_readme()
 {
if (!ctx.cfg.root_readme)
return;
-   if (ctx.cfg.about_filter)
-   cgit_open_filter(ctx.cfg.about_filter, ctx.cfg.root_readme);
+   cgit_open_filter(ctx.cfg.about_filter, ctx.cfg.root_readme);
html_include(ctx.cfg.root_readme);
-   if (ctx.cfg.about_filter)
-   cgit_close_filter(ctx.cfg.about_filter);
+   cgit_close_filter(ctx.cfg.about_filter);
 }
diff --git a/ui-summary.c b/ui-summary.c
index 725f3ab..ddd8f1b 100644
--- a/ui-summary.c
+++ b/ui-summary.c
@@ -151,16 +151,12 @@ void cgit_print_repo_readme(char *path)
 * filesystem, while applying the about-filter.
 */
html("");
-   if (ctx.repo->about_filter)
-   cgit_open_filter(ctx.repo->about_filter, filename);
-
+   cgit_open_filter(ctx.repo->about_filter, filename);
if (ref)
cgit_print_file(filename, ref, 1);
else
html_include(filename);
-
-   if (ctx.repo->about_filter)
-   cgit_close_filter(ctx.repo->about_filter);
+   cgit_close_filter(ctx.repo->about_filter);
 
html("");
if (free_filename)
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v2 3/9] filter: introduce "filter type" prefix

[Jason A. Donenfeld]

From: John Keeping 

This allows different filter implementations to be specified in the
configuration file.  Currently only "exec" is supported, but it may now
be specified either with or without the "exec:" prefix.

Signed-off-by: John Keeping 
---
 cgitrc.5.txt |  9 +
 filter.c | 33 +++--
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/cgitrc.5.txt b/cgitrc.5.txt
index 52caed0..60159f6 100644
--- a/cgitrc.5.txt
+++ b/cgitrc.5.txt
@@ -557,6 +557,15 @@ config files, e.g. "repo.desc" becomes "desc".
 
 FILTER API
 --
+By default, filters are separate processes that are executed each time they
+are needed.  Alternative technologies may be used by prefixing the filter
+specification with the relevant string; available values are:
+
+'exec:'::
+   The default "one process per filter" mode.
+
+Parameters are provided to filters as follows.
+
 about filter::
This filter is given a single parameter: the filename of the source
file to filter. The filter can use the filename to determine (for
diff --git a/filter.c b/filter.c
index 0f3edb0..ba66e46 100644
--- a/filter.c
+++ b/filter.c
@@ -64,7 +64,7 @@ done:
 static void fprintf_exec_filter(struct cgit_filter *base, FILE *f, const char 
*prefix)
 {
struct cgit_exec_filter *filter = (struct cgit_exec_filter *) base;
-   fprintf(f, "%s%s\n", prefix, filter->cmd);
+   fprintf(f, "%sexec:%s\n", prefix, filter->cmd);
 }
 
 int cgit_open_filter(struct cgit_filter *filter, ...)
@@ -125,10 +125,39 @@ static struct cgit_filter *new_exec_filter(const char 
*cmd, filter_type filterty
return &f->base;
 }
 
+static const struct {
+   const char *prefix;
+   struct cgit_filter *(*ctor)(const char *cmd, filter_type filtertype);
+} filter_specs[] = {
+   { "exec", new_exec_filter },
+};
+
 struct cgit_filter *cgit_new_filter(const char *cmd, filter_type filtertype)
 {
+   char *colon;
+   int i;
+   size_t len;
if (!cmd || !cmd[0])
return NULL;
 
-   return new_exec_filter(cmd, filtertype);
+   colon = strchr(cmd, ':');
+   len = colon - cmd;
+   /*
+* In case we're running on Windows, don't allow a single letter before
+* the colon.
+*/
+   if (len == 1)
+   colon = NULL;
+
+   /* If no prefix is given, exec filter is the default. */
+   if (!colon)
+   return new_exec_filter(cmd, filtertype);
+
+   for (i = 0; i < ARRAY_SIZE(filter_specs); i++) {
+   if (len == strlen(filter_specs[i].prefix) &&
+   !strncmp(filter_specs[i].prefix, cmd, len))
+   return filter_specs[i].ctor(colon + 1, filtertype);
+   }
+
+   die("Invalid filter type: %.*s", (int) len, cmd);
 }
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v2 9/9] filter: add gravatar scripts

[Jason A. Donenfeld]

The lua one is hugely faster than the python one, but both are included
for comparison.

Signed-off-by: Jason A. Donenfeld 
---
 filters/email-gravatar.lua | 25 +
 filters/email-gravatar.py  | 33 +
 2 files changed, 58 insertions(+)
 create mode 100644 filters/email-gravatar.lua
 create mode 100755 filters/email-gravatar.py

diff --git a/filters/email-gravatar.lua b/filters/email-gravatar.lua
new file mode 100644
index 000..ef1bdbc
--- /dev/null
+++ b/filters/email-gravatar.lua
@@ -0,0 +1,25 @@
+-- This script may be used with the email-filter or repo.email-filter settings 
in cgitrc.
+-- It adds gravatar icons to author names. It is designed to be used with the 
lua:
+-- prefix in filters. It is much faster than the corresponding python script.
+--
+-- Requirements:
+-- luacrypto >= 0.3
+-- 
+--
+
+require("crypto")
+
+function filter_open(email)
+   buffer = ""
+   md5 = crypto.digest("md5", email:sub(2, -2):lower())
+end
+
+function filter_close()
+   html(" " .. buffer)
+end
+
+function filter_write(str)
+   buffer = buffer .. str
+end
+
+
diff --git a/filters/email-gravatar.py b/filters/email-gravatar.py
new file mode 100755
index 000..52a184b
--- /dev/null
+++ b/filters/email-gravatar.py
@@ -0,0 +1,33 @@
+#!/usr/bin/env python3
+
+# Please prefer the email-gravatar.lua using lua: as a prefix over this 
script. This
+# script is very slow, in comparison.
+#
+# This script may be used with the email-filter or repo.email-filter settings 
in cgitrc.
+#
+# The following environment variables can be used to retrieve the configuration
+# of the repository for which this script is called:
+# CGIT_REPO_URL( = repo.url   setting )
+# CGIT_REPO_NAME   ( = repo.name  setting )
+# CGIT_REPO_PATH   ( = repo.path  setting )
+# CGIT_REPO_OWNER  ( = repo.owner setting )
+# CGIT_REPO_DEFBRANCH  ( = repo.defbranch setting )
+# CGIT_REPO_SECTION( = sectionsetting )
+# CGIT_REPO_CLONE_URL  ( = repo.clone-url setting )
+#
+# It receives an email address on argv[1] and text on stdin. It prints
+# to stdout that text prepended by a gravatar at 10pt.
+
+import sys
+import hashlib
+
+email = sys.argv[1].lower().strip()
+if email[0] == '<':
+email = email[1:]
+if email[-1] == '>':
+email = email[0:-1]
+
+md5 = hashlib.md5(email.encode()).hexdigest()
+text = sys.stdin.read().strip()
+
+print(" " + text)
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v2 5/9] filter: basic write hooking infrastructure

[Jason A. Donenfeld]

Filters can now call hook_write and unhook_write if they want to
redirect writing to stdout to a different function. This saves us from
potential file descriptor pipes and other less efficient mechanisms.

We do this instead of replacing the call in html_raw because some places
stdlib's printf functions are used (ui-patch or within git itself),
which has its own internal buffering, which makes it difficult to
interlace our function calls. So, we dlsym libc's write and then
override it in the link stage.

While we're at it, we move considerations of argument count into the
generic new filter handler.

Signed-off-by: Jason A. Donenfeld 
---
 cgit.c   |  2 ++
 cgit.h   |  3 ++-
 cgit.mk  |  4 +++-
 filter.c | 81 +++-
 4 files changed, 67 insertions(+), 23 deletions(-)

diff --git a/cgit.c b/cgit.c
index 4f31e58..725fd65 100644
--- a/cgit.c
+++ b/cgit.c
@@ -904,6 +904,8 @@ int main(int argc, const char **argv)
const char *path;
int err, ttl;
 
+   cgit_init_filters();
+
prepare_context(&ctx);
cgit_repolist.length = 0;
cgit_repolist.count = 0;
diff --git a/cgit.h b/cgit.h
index 893c38f..519d2af 100644
--- a/cgit.h
+++ b/cgit.h
@@ -61,13 +61,13 @@ struct cgit_filter {
int (*close)(struct cgit_filter *);
void (*fprintf)(struct cgit_filter *, FILE *, const char *prefix);
void (*cleanup)(struct cgit_filter *);
+   int argument_count;
 };
 
 struct cgit_exec_filter {
struct cgit_filter base;
char *cmd;
char **argv;
-   int extra_args;
int old_stdout;
int pipe_fh[2];
int pid;
@@ -357,6 +357,7 @@ extern void cgit_fprintf_filter(struct cgit_filter *filter, 
FILE *f, const char
 extern void cgit_exec_filter_init(struct cgit_exec_filter *filter, char *cmd, 
char **argv);
 extern struct cgit_filter *cgit_new_filter(const char *cmd, filter_type 
filtertype);
 extern void cgit_cleanup_filters(void);
+extern void cgit_init_filters(void);
 
 extern void cgit_prepare_repo_env(struct cgit_repo * repo);
 
diff --git a/cgit.mk b/cgit.mk
index 19a76e7..9d6dea8 100644
--- a/cgit.mk
+++ b/cgit.mk
@@ -61,6 +61,8 @@ $(CGIT_VERSION_OBJS): $(CGIT_PREFIX)VERSION
 $(CGIT_VERSION_OBJS): EXTRA_CPPFLAGS = \
-DCGIT_VERSION='"$(CGIT_VERSION)"'
 
+CGIT_LIBS += -ldl
+
 
 # Git handles dependencies using ":=" so dependencies in CGIT_OBJ are not
 # handled by that and we must handle them ourselves.
@@ -88,4 +90,4 @@ $(CGIT_OBJS): %.o: %.c GIT-CFLAGS $(CGIT_PREFIX)CGIT-CFLAGS 
$(missing_dep_dirs)
$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) 
$(CGIT_CFLAGS) $<
 
 $(CGIT_PREFIX)cgit: $(CGIT_OBJS) GIT-LDFLAGS $(GITLIBS)
-   $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) 
$(LIBS)
+   $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) 
$(LIBS) $(CGIT_LIBS)
diff --git a/filter.c b/filter.c
index 30bc74b..f5a5992 100644
--- a/filter.c
+++ b/filter.c
@@ -12,6 +12,11 @@
 #include 
 #include 
 #include 
+#include 
+
+static ssize_t (*libc_write)(int fd, const void *buf, size_t count);
+static ssize_t (*filter_write)(struct cgit_filter *base, const void *buf, 
size_t count) = NULL;
+static struct cgit_filter *current_write_filter = NULL;
 
 static inline void reap_filter(struct cgit_filter *filter)
 {
@@ -32,12 +37,43 @@ void cgit_cleanup_filters(void)
}
 }
 
+void cgit_init_filters(void)
+{
+   libc_write = dlsym(RTLD_NEXT, "write");
+   if (!libc_write)
+   die("Could not locate libc's write function");
+}
+
+ssize_t write(int fd, const void *buf, size_t count)
+{
+   if (fd != STDOUT_FILENO || !filter_write)
+   return libc_write(fd, buf, count);
+   return filter_write(current_write_filter, buf, count);
+}
+
+static inline void hook_write(struct cgit_filter *filter, ssize_t 
(*new_write)(struct cgit_filter *base, const void *buf, size_t count))
+{
+   /* We want to avoid buggy nested patterns. */
+   assert(filter_write == NULL);
+   assert(current_write_filter == NULL);
+   current_write_filter = filter;
+   filter_write = new_write;
+}
+
+static inline void unhook_write()
+{
+   assert(filter_write != NULL);
+   assert(current_write_filter != NULL);
+   filter_write = NULL;
+   current_write_filter = NULL;
+}
+
 static int open_exec_filter(struct cgit_filter *base, va_list ap)
 {
struct cgit_exec_filter *filter = (struct cgit_exec_filter *) base;
int i;
 
-   for (i = 0; i < filter->extra_args; i++)
+   for (i = 0; i < filter->base.argument_count; i++)
filter->argv[i+1] = va_arg(ap, char *);
 
filter->old_stdout = chk_positive(dup(STDOUT_FILENO),
@@ -74,7 +110,7 @@ static int close_exec_filter(struct cgit_filter *base)
die("Subprocess %s exited abnormally", filter->cmd);
 
 done:
-   for (i = 0; i < filter->extra_args; i++)
+   for (i

[PATCH v2 4/9] filter: allow for cleanup hook for filter types

[Jason A. Donenfeld]

At some point, we're going to want to do lazy deallocation of filters.
For example, if we implement lua, we'll want to load the lua runtime
once for each filter, even if that filter is called many times.
Similarly, for persistent exec filters, we'll want to load it once,
despite many open_filter and close_filter calls, and only reap the child
process at the end of the cgit process. For this reason, we add here a
cleanup function that is called at the end of cgit's main().

Signed-off-by: Jason A. Donenfeld 
---
 cgit.c   |  1 +
 cgit.h   |  2 ++
 filter.c | 90 +---
 3 files changed, 66 insertions(+), 27 deletions(-)

diff --git a/cgit.c b/cgit.c
index 29b658e..4f31e58 100644
--- a/cgit.c
+++ b/cgit.c
@@ -951,6 +951,7 @@ int main(int argc, const char **argv)
ctx.cfg.cache_size = 0;
err = cache_process(ctx.cfg.cache_size, ctx.cfg.cache_root,
ctx.qry.raw, ttl, process_request, &ctx);
+   cgit_cleanup_filters();
if (err)
cgit_print_error("Error processing page: %s (%d)",
 strerror(err), err);
diff --git a/cgit.h b/cgit.h
index 92e8c55..893c38f 100644
--- a/cgit.h
+++ b/cgit.h
@@ -60,6 +60,7 @@ struct cgit_filter {
int (*open)(struct cgit_filter *, va_list ap);
int (*close)(struct cgit_filter *);
void (*fprintf)(struct cgit_filter *, FILE *, const char *prefix);
+   void (*cleanup)(struct cgit_filter *);
 };
 
 struct cgit_exec_filter {
@@ -355,6 +356,7 @@ extern int cgit_close_filter(struct cgit_filter *filter);
 extern void cgit_fprintf_filter(struct cgit_filter *filter, FILE *f, const 
char *prefix);
 extern void cgit_exec_filter_init(struct cgit_exec_filter *filter, char *cmd, 
char **argv);
 extern struct cgit_filter *cgit_new_filter(const char *cmd, filter_type 
filtertype);
+extern void cgit_cleanup_filters(void);
 
 extern void cgit_prepare_repo_env(struct cgit_repo * repo);
 
diff --git a/filter.c b/filter.c
index ba66e46..30bc74b 100644
--- a/filter.c
+++ b/filter.c
@@ -13,6 +13,25 @@
 #include 
 #include 
 
+static inline void reap_filter(struct cgit_filter *filter)
+{
+   if (filter && filter->cleanup)
+   filter->cleanup(filter);
+}
+
+void cgit_cleanup_filters(void)
+{
+   int i;
+   reap_filter(ctx.cfg.about_filter);
+   reap_filter(ctx.cfg.commit_filter);
+   reap_filter(ctx.cfg.source_filter);
+   for (i = 0; i < cgit_repolist.count; ++i) {
+   reap_filter(cgit_repolist.repos[i].about_filter);
+   reap_filter(cgit_repolist.repos[i].commit_filter);
+   reap_filter(cgit_repolist.repos[i].source_filter);
+   }
+}
+
 static int open_exec_filter(struct cgit_filter *base, va_list ap)
 {
struct cgit_exec_filter *filter = (struct cgit_exec_filter *) base;
@@ -67,34 +86,17 @@ static void fprintf_exec_filter(struct cgit_filter *base, 
FILE *f, const char *p
fprintf(f, "%sexec:%s\n", prefix, filter->cmd);
 }
 
-int cgit_open_filter(struct cgit_filter *filter, ...)
-{
-   int result;
-   va_list ap;
-   va_start(ap, filter);
-   result = filter->open(filter, ap);
-   va_end(ap);
-   return result;
-}
-
-int cgit_close_filter(struct cgit_filter *filter)
-{
-   return filter->close(filter);
-}
-
-void cgit_fprintf_filter(struct cgit_filter *filter, FILE *f, const char 
*prefix)
-{
-   filter->fprintf(filter, f, prefix);
-}
-
-void cgit_exec_filter_init(struct cgit_exec_filter *filter, char *cmd, char 
**argv)
+static void cleanup_exec_filter(struct cgit_filter *base)
 {
-   memset(filter, 0, sizeof(*filter));
-   filter->base.open = open_exec_filter;
-   filter->base.close = close_exec_filter;
-   filter->base.fprintf = fprintf_exec_filter;
-   filter->cmd = cmd;
-   filter->argv = argv;
+   struct cgit_exec_filter *filter = (struct cgit_exec_filter *) base;
+   if (filter->argv) {
+   free(filter->argv);
+   filter->argv = NULL;
+   }
+   if (filter->cmd) {
+   free(filter->cmd);
+   filter->cmd = NULL;
+   }
 }
 
 static struct cgit_filter *new_exec_filter(const char *cmd, filter_type 
filtertype)
@@ -125,6 +127,39 @@ static struct cgit_filter *new_exec_filter(const char 
*cmd, filter_type filterty
return &f->base;
 }
 
+void cgit_exec_filter_init(struct cgit_exec_filter *filter, char *cmd, char 
**argv)
+{
+   memset(filter, 0, sizeof(*filter));
+   filter->base.open = open_exec_filter;
+   filter->base.close = close_exec_filter;
+   filter->base.fprintf = fprintf_exec_filter;
+   filter->base.cleanup = cleanup_exec_filter;
+   filter->cmd = cmd;
+   filter->argv = argv;
+}
+
+int cgit_open_filter(struct cgit_filter *filter, ...)
+{
+   int result;
+   va_list ap;
+   va_start(ap, filter);
+   result = filter->open(filter, ap);
+   va_end(ap)

[PATCH v2 0/9] filter framework and lua support: complete

[Jason A. Donenfeld]

In this second installment I've squashed things down and cleanly separated
the framework commits from the lua commits. It should be much easier to
comprehend.

The weakest part is still the Makefile changes, which might not be very
portable and could probably be more configurable somehow. If anybody would
like to step in and brush this up, that'd be quite nice. You can base commits
off of jd-jk/filter-infra, and I'll fold it into this series.

The other controversial item is the write() hooking. The reason we do this
instead of replacing the html_raw function is that in various places, such
as lines 82 and 83 of ui-patch.c, we use other facilities for writing to
standard out -- either via stdio (which has its own buffering), or via git,
which also has its own buffering. So, it turned out to be safer to hook
libc's write than to try and monkey patch git itself.

Another comment brought up in v1 was adding a filter_lua.c file. I'd like to
keep things fairly compact for now, partially because I like inline and
static functions, and partially because I don't want to create a filter.h.
Right now despite supporting a scripting language, this is pretty lean; I'd
like for it to remain so. At least for now.

Jason A. Donenfeld (6):
  filter: allow for cleanup hook for filter types
  filter: basic write hooking infrastructure
  filter: add lua support
  filter: return on null filter from open and close
  filter: add support for email filter
  filter: add gravatar scripts

John Keeping (3):
  filter: add fprintf_filter function
  filter: add interface layer
  filter: introduce "filter type" prefix

 cgit.c |  15 +-
 cgit.h |  18 ++-
 cgit.mk|  22 ++-
 cgitrc.5.txt   |  56 +++
 filter.c   | 376 ++---
 filters/email-gravatar.lua |  25 +++
 filters/email-gravatar.py  |  33 
 shared.c   |   1 +
 ui-commit.c|  22 ++-
 ui-log.c   |   2 +
 ui-refs.c  |   9 +-
 ui-repolist.c  |   6 +-
 ui-snapshot.c  |  11 +-
 ui-summary.c   |   8 +-
 ui-tag.c   |   2 +
 15 files changed, 550 insertions(+), 56 deletions(-)
 create mode 100644 filters/email-gravatar.lua
 create mode 100755 filters/email-gravatar.py

-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v2 1/9] filter: add fprintf_filter function

[Jason A. Donenfeld]

From: John Keeping 

This stops the code in cgit.c::print_repo needing to inspect the
cgit_filter structure, meaning that we can abstract out different filter
types that will have different fields that need to be printed.

Signed-off-by: John Keeping 
---
 cgit.c   | 6 +++---
 cgit.h   | 1 +
 filter.c | 5 +
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/cgit.c b/cgit.c
index 0be41b8..29b658e 100644
--- a/cgit.c
+++ b/cgit.c
@@ -706,11 +706,11 @@ static void print_repo(FILE *f, struct cgit_repo *repo)
fprintf(f, "repo.enable-log-linecount=%d\n",
repo->enable_log_linecount);
if (repo->about_filter && repo->about_filter != ctx.cfg.about_filter)
-   fprintf(f, "repo.about-filter=%s\n", repo->about_filter->cmd);
+   cgit_fprintf_filter(repo->about_filter, f, 
"repo.about-filter=");
if (repo->commit_filter && repo->commit_filter != ctx.cfg.commit_filter)
-   fprintf(f, "repo.commit-filter=%s\n", repo->commit_filter->cmd);
+   cgit_fprintf_filter(repo->commit_filter, f, 
"repo.commit-filter=");
if (repo->source_filter && repo->source_filter != ctx.cfg.source_filter)
-   fprintf(f, "repo.source-filter=%s\n", repo->source_filter->cmd);
+   cgit_fprintf_filter(repo->source_filter, f, 
"repo.source-filter=");
if (repo->snapshots != ctx.cfg.snapshots) {
char *tmp = build_snapshot_setting(repo->snapshots);
fprintf(f, "repo.snapshots=%s\n", tmp ? tmp : "");
diff --git a/cgit.h b/cgit.h
index e6e7715..9b4be26 100644
--- a/cgit.h
+++ b/cgit.h
@@ -345,6 +345,7 @@ extern int cgit_parse_snapshots_mask(const char *str);
 
 extern int cgit_open_filter(struct cgit_filter *filter, ...);
 extern int cgit_close_filter(struct cgit_filter *filter);
+extern void cgit_fprintf_filter(struct cgit_filter *filter, FILE *f, const 
char *prefix);
 extern struct cgit_filter *cgit_new_filter(const char *cmd, filter_type 
filtertype);
 
 extern void cgit_prepare_repo_env(struct cgit_repo * repo);
diff --git a/filter.c b/filter.c
index d8c0116..80cf689 100644
--- a/filter.c
+++ b/filter.c
@@ -63,6 +63,11 @@ done:
 
 }
 
+void cgit_fprintf_filter(struct cgit_filter *filter, FILE *f, const char 
*prefix)
+{
+   fprintf(f, "%s%s\n", prefix, filter->cmd);
+}
+
 struct cgit_filter *cgit_new_filter(const char *cmd, filter_type filtertype)
 {
struct cgit_filter *f;
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

[PATCH v2] filters: Improved syntax-highlighting.py

[Stefan Tatschner]

- Switched back to python2 according to a problem in pygments with python3.
  With the next release of pygments this problem should be fixed.
  Issue see here:
  https://bitbucket.org/birkenfeld/pygments-main/issue/901/problems-with-python3
- Just read the stdin, decode it to utf-8 and ignore unknown signs. This ensures
  that even destroyed files do not cause any errors in the filter.
- Improved language guessing:
  -> At first use guess_lexer_for_filename for a better detection of the used
 programming languages (even mixed cases will be detected, e.g. php + html).
  -> If nothing was found look if there is a shebang and use guess_lexer.
  -> As default/fallback choose TextLexer.
- Using inline CSS instead of this sys.stdout.print() hack.
- Using trac theme for pygments (it is very clean and not intrusive like the
  default or pastie theme).
- I had to fix cgit.css according to a alignment issue with the line-numbers
  table.

Signed-off-by: Stefan Tatschner 
---
 cgit.css   |  1 +
 filters/syntax-highlighting.py | 48 +-
 2 files changed, 30 insertions(+), 19 deletions(-)

diff --git a/cgit.css b/cgit.css
index 71b0b9b..ef99b5d 100644
--- a/cgit.css
+++ b/cgit.css
@@ -289,6 +289,7 @@ div#cgit table.blob td.linenumbers {
 
 div#cgit table.blob pre {
padding: 0; margin: 0;
+   line-height: 125%;
 }
 
 div#cgit table.blob td.linenumbers a,
diff --git a/filters/syntax-highlighting.py b/filters/syntax-highlighting.py
index 72d9097..b95baed 100755
--- a/filters/syntax-highlighting.py
+++ b/filters/syntax-highlighting.py
@@ -1,13 +1,16 @@
-#!/usr/bin/env python3
+#!/usr/bin/env python2
 
-# This script uses Pygments and Python3. You must have both installed for this 
to work.
+# This script uses Pygments and Python2. You must have both installed
+# for this to work.
+#
 # http://pygments.org/
 # http://python.org/
 #
-# It may be used with the source-filter or repo.source-filter settings in 
cgitrc.
+# It may be used with the source-filter or repo.source-filter settings
+# in cgitrc.
 #
-# The following environment variables can be used to retrieve the configuration
-# of the repository for which this script is called:
+# The following environment variables can be used to retrieve the
+# configuration of the repository for which this script is called:
 # CGIT_REPO_URL( = repo.url   setting )
 # CGIT_REPO_NAME   ( = repo.name  setting )
 # CGIT_REPO_PATH   ( = repo.path  setting )
@@ -18,22 +21,29 @@
 
 
 import sys
-import cgi
-import codecs
-from pygments.lexers import get_lexer_for_filename
 from pygments import highlight
+from pygments.util import ClassNotFound
+from pygments.lexers import TextLexer
+from pygments.lexers import guess_lexer
+from pygments.lexers import guess_lexer_for_filename
 from pygments.formatters import HtmlFormatter
 
-sys.stdin = codecs.getreader("utf-8")(sys.stdin.detach())
-sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())
-doc = sys.stdin.read()
+
+# read stdin and decode to utf-8. ignore any unkown signs.
+data = sys.stdin.read().decode(encoding='utf-8', errors='ignore')
+filename = sys.argv[1]
+formatter = HtmlFormatter(encoding='utf-8', style='trac', noclasses=True)
+
 try:
-   lexer = get_lexer_for_filename(sys.argv[1])
-   formatter = HtmlFormatter(style='pastie')
-   sys.stdout.write("")
-   sys.stdout.write(formatter.get_style_defs('.highlight'))
-   sys.stdout.write("")
+lexer = guess_lexer_for_filename(filename, data, encoding='utf-8')
+except ClassNotFound:
+# check if there is any shebang
+if data[0:2] == '#!':
+lexer = guess_lexer(data, encoding='utf-8')
+else:
+lexer = TextLexer(encoding='utf-8')
+except TypeError:
+lexer = TextLexer(encoding='utf-8')
 
-   highlight(doc, lexer, formatter, sys.stdout)
-except:
-   sys.stdout.write(str(cgi.escape(doc).encode("ascii", 
"xmlcharrefreplace"), "ascii"))
+# highlight! :-)
+highlight(data, lexer, formatter, outfile=sys.stdout)
-- 
1.8.5.2

___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH 06/12] filter: add preliminary lua support

[Florian Pritz]

On 13.01.2014 05:11, Jason A. Donenfeld wrote:
> Signed-off-by: Jason A. Donenfeld 
> ---
>  cgit.h   |   2 +-
>  cgit.mk  |  13 ++-
>  filter.c | 284 
> ---

All those *_lua_filter functions look rather self contained, maybe they
should be split into filter_lua.c for readability?

Also I'm not sure why documenting the filter and adding error reporting
to totally new code go into separate commits. They are one logical
change after all.

As Lukas already said it's probably a lot better if you split the
cleanup and moving-stuff-around into it's own patch. (Or merge it into
"basic write hooking infrastructure" since you create that code there so
it should be created at the correct place)




signature.asc
Description: OpenPGP digital signature
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH 06/12] filter: add preliminary lua support

[John Keeping]

On Mon, Jan 13, 2014 at 05:11:13AM +0100, Jason A. Donenfeld wrote:
[snip]
> +static int html_lua_filter(lua_State *lua_state)
> +{
> + size_t len;
> + const char *str;
> +
> + str = lua_tostring(lua_state, 1);
> + if (!str)
> + return 0;
> + len = strlen(str);
> + libc_write(STDOUT_FILENO, str, len);
> + return 0;
> +}
> +
> +static void cleanup_lua_filter(struct cgit_filter *base)
> +{
> + struct lua_filter *filter = (struct lua_filter *) base;
> +
> + if (!filter->lua_state)
> + return;
> +
> + lua_close(filter->lua_state);
> + filter->lua_state = NULL;
> + if (filter->script_file) {
> + free(filter->script_file);
> + filter->script_file = NULL;
> + }
> +}
> +
> +static int init_lua_filter(struct lua_filter *filter)
> +{
> + if (filter->lua_state)
> + return 0;
> +
> + if (!(filter->lua_state = luaL_newstate()))
> + return 1;
> +
> + luaL_openlibs(filter->lua_state);
> +
> + lua_pushcfunction(filter->lua_state, html_lua_filter);
> + lua_setglobal(filter->lua_state, "html");

It would be good to provide some of the other html_* functions here,
particularly html_txt so that filters don't need to re-invent the wheel
for escaping output.  That's probably slightly harder than the plain
HTML, but I suspect we just need to wrap the call to the underlying
function in an unhook_write/hook_write pair.

> +
> + if (luaL_dofile(filter->lua_state, filter->script_file)) {
> + lua_close(filter->lua_state);
> + filter->lua_state = NULL;
> + return 1;
> + }
> + return 0;
> +}
> +
> +static int open_lua_filter(struct cgit_filter *base, va_list ap)
> +{
> + struct lua_filter *filter = (struct lua_filter *) base;
> + int i;
> +
> + if (init_lua_filter(filter))
> + return 1;
> +
> + hook_write(base, write_lua_filter);
> +
> + lua_getglobal(filter->lua_state, "filter_open");
> + for (i = 0; i < filter->base.argument_count; ++i)
> + lua_pushstring(filter->lua_state, va_arg(ap, char *));
> + if (lua_pcall(filter->lua_state, filter->base.argument_count, 0, 0))
> + return 1;
> + return 0;
> +}
> +
> +static int close_lua_filter(struct cgit_filter *base)
> +{
> + struct lua_filter *filter = (struct lua_filter *) base;
> + int ret = 0;
> +
> + lua_getglobal(filter->lua_state, "filter_close");
> + if (lua_pcall(filter->lua_state, 0, 0, 0))
> + ret = 1;
> + unhook_write();
> + return ret;
> +}
> +
> +static void fprintf_lua_filter(struct cgit_filter *base, FILE *f, const char 
> *prefix)
> +{
> + struct lua_filter *filter = (struct lua_filter *) base;
> + fprintf(f, "%slua:%s\n", prefix, filter->script_file);
> +}
> +
> +
> +static struct cgit_filter *new_lua_filter(const char *cmd, int 
> argument_count)
> +{
> + struct lua_filter *filter;
> +
> + filter = xmalloc(sizeof(*filter));
> + memset(filter, 0, sizeof(*filter));
> + filter->base.open = open_lua_filter;
> + filter->base.close = close_lua_filter;
> + filter->base.fprintf = fprintf_lua_filter;
> + filter->base.cleanup = cleanup_lua_filter;
> + filter->base.argument_count = argument_count;
> + filter->script_file = xstrdup(cmd);
> +
> + return &filter->base;
>  }
>  
> +#endif
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH 06/12] filter: add preliminary lua support

[John Keeping]

On Mon, Jan 13, 2014 at 09:31:39AM +0100, Lukas Fleischer wrote:
> This patch is quite messy and hard to read. I read your cover-letter but
> maybe you still want to clean this up when dealing with the other
> suggestions during a rebase -- shouldn't be too hard when using an
> editor with good Git integration (like fugitive for Vim).
> 
> On Mon, 13 Jan 2014 at 05:11:13, Jason A. Donenfeld wrote:
> > [...]
> > +ifdef NO_LUA
> 
> We should document this in the installation instructions section in the
> README. I also wonder whether this should made an opt-in feature?
> 
> > +   CFLAGS += -DNO_LUA
> > +else
> > +   CGIT_LIBS += -llua
> 
> Similar: Add Lua/LuaJIT (Will you squash the LuaJIT Makefile fix into
> this one? Or is there any reason to use Lua first and switch to LuaJIT
> later?) to the dependencies section of the README file and mention that
> it is optional.

I think we should support both vanilla Lua and LuaJIT if we can (I
believe LuaJIT can be used as a drop-in replacement, so there's no
reason this shouldn't be possible).

> > +endif
> > +
> > +CGIT_LIBS += -ldl
> > +
> > +
> > +
> >  CGIT_OBJ_NAMES += cgit.o
> >  CGIT_OBJ_NAMES += cache.o
> >  CGIT_OBJ_NAMES += cmd.o
> > [...]
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH 06/12] filter: add preliminary lua support

[Lukas Fleischer]

On Mon, 13 Jan 2014 at 05:11:13, Jason A. Donenfeld wrote:
> Signed-off-by: Jason A. Donenfeld 
> ---
>  cgit.h   |   2 +-
>  cgit.mk  |  13 ++-
>  filter.c | 284 
> ---
>  3 files changed, 230 insertions(+), 69 deletions(-)
> 
> [...]
> +static ssize_t write_lua_filter(struct cgit_filter *base, const void *buf, 
> size_t count)
> +{
> +   struct lua_filter *filter = (struct lua_filter *) base;
> +
> +   lua_getglobal(filter->lua_state, "filter_write");
> +   lua_pushlstring(filter->lua_state, buf, count);
> +   if (lua_pcall(filter->lua_state, 1, 0, 0)) {
> +   errno = EPROTO;

Just noticed that this also breaks compilation under OpenBSD:

../filter.c:199: error: 'EPROTO' undeclared (first use in this function)

The error number seems to be part of POSIX, though, so not sure if we
should care about OpenBSD being non-compliant here.

> +   return -1;
> +   }
> +   return count;
> +}
> [...]
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH 06/12] filter: add preliminary lua support

[Lukas Fleischer]

This patch is quite messy and hard to read. I read your cover-letter but
maybe you still want to clean this up when dealing with the other
suggestions during a rebase -- shouldn't be too hard when using an
editor with good Git integration (like fugitive for Vim).

On Mon, 13 Jan 2014 at 05:11:13, Jason A. Donenfeld wrote:
> [...]
> +ifdef NO_LUA

We should document this in the installation instructions section in the
README. I also wonder whether this should made an opt-in feature?

> +   CFLAGS += -DNO_LUA
> +else
> +   CGIT_LIBS += -llua

Similar: Add Lua/LuaJIT (Will you squash the LuaJIT Makefile fix into
this one? Or is there any reason to use Lua first and switch to LuaJIT
later?) to the dependencies section of the README file and mention that
it is optional.

> +endif
> +
> +CGIT_LIBS += -ldl
> +
> +
> +
>  CGIT_OBJ_NAMES += cgit.o
>  CGIT_OBJ_NAMES += cache.o
>  CGIT_OBJ_NAMES += cmd.o
> [...]
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit

Re: [PATCH 05/12] filter: basic write hooking infrastructure

[Lukas Fleischer]

On Mon, 13 Jan 2014 at 05:11:12, Jason A. Donenfeld wrote:
> Filters can now call hook_write and unhook_write if they want to
> redirect writing to stdout to a different function. This saves us from
> potential file descriptor pipes and other less efficient mechanisms.
> 
> We do this instead of replacing the call in html_raw because some places
> stdlib's printf functions are used (ui-patch or within git itself),
> which has its own internal buffering, which makes it difficult to
> interlace our function calls. So, we dlsym libc's write and then
> override it in the link stage.

Clever. Not sure whether I like it, though. I think it would be much
better to have a more transparent hooking mechanism that does use any
"tricks". Could you elaborate on where Git directly prints for us?

> 
> Signed-off-by: Jason A. Donenfeld 
> ---
>  cgit.c   |  2 ++
>  cgit.h   |  1 +
>  cgit.mk  |  4 +++-
>  filter.c | 30 ++
>  4 files changed, 36 insertions(+), 1 deletion(-)
> 
> [...]
> diff --git a/cgit.mk b/cgit.mk
> index 19a76e7..9d6dea8 100644
> --- a/cgit.mk
> +++ b/cgit.mk
> @@ -61,6 +61,8 @@ $(CGIT_VERSION_OBJS): $(CGIT_PREFIX)VERSION
>  $(CGIT_VERSION_OBJS): EXTRA_CPPFLAGS = \
> -DCGIT_VERSION='"$(CGIT_VERSION)"'
>  
> +CGIT_LIBS += -ldl

This breaks compilation at least on FreeBSD and OpenBSD which do no have
libdl (and have dlsym() built-in as a part of libc). I am not sure about
other platforms. So if we go this way, we should check how to make this
cross-platform compatible.

> +

Stray newline?

>  
>  # Git handles dependencies using ":=" so dependencies in CGIT_OBJ are not
>  # handled by that and we must handle them ourselves.
> @@ -88,4 +90,4 @@ $(CGIT_OBJS): %.o: %.c GIT-CFLAGS $(CGIT_PREFIX)CGIT-CFLAGS 
> $(missing_dep_dirs)
> $(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(ALL_CFLAGS) 
> $(EXTRA_CPPFLAGS) $(CGIT_CFLAGS) $<
>  
>  $(CGIT_PREFIX)cgit: $(CGIT_OBJS) GIT-LDFLAGS $(GITLIBS)
> -   $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter 
> %.o,$^) $(LIBS)
> +   $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter 
> %.o,$^) $(LIBS) $(CGIT_LIBS)
> diff --git a/filter.c b/filter.c
> index 86c1d5d..8990575 100644
> --- a/filter.c
> +++ b/filter.c
> @@ -12,6 +12,10 @@
>  #include 
>  #include 
>  #include 
> +#include 
> +
> +static ssize_t (*libc_write)(int fd, const void *buf, size_t count);
> +static ssize_t (*filter_write)(const void *buf, size_t count) = NULL;
>  
>  static int open_exec_filter(struct cgit_filter *base, va_list ap)
>  {
> @@ -192,3 +196,29 @@ void cgit_cleanup_filters(void)
> reap_filter(cgit_repolist.repos[i].source_filter);
> }
>  }
> +
> +void cgit_init_filters(void)
> +{
> +   libc_write = dlsym(RTLD_NEXT, "write");
> +   if (!libc_write)
> +   die("Could not locate libc's write function");
> +}
> +
> +ssize_t write(int fd, const void *buf, size_t count)
> +{
> +   if (fd != STDOUT_FILENO || !filter_write)
> +   return libc_write(fd, buf, count);
> +   return filter_write(buf, count);
> +}
> +
> +static inline void hook_write(ssize_t (*new_write)(const void *buf, size_t 
> count))
> +{
> +   /* We want to avoid buggy nested patterns. */
> +   assert(filter_write == NULL);
> +   filter_write = new_write;
> +}
> +static inline void unhook_write()
> +{
> +   assert(filter_write != NULL);
> +   filter_write = NULL;
> +}
> -- 
> 1.8.5.2
> 
> ___
> CGit mailing list
> CGit@lists.zx2c4.com
> http://lists.zx2c4.com/mailman/listinfo/cgit
___
CGit mailing list
CGit@lists.zx2c4.com
http://lists.zx2c4.com/mailman/listinfo/cgit