The following plain tex document $$ n \choose l $$ \bye
worked in 0.80 as in pdftex. I'm not sure quite which version it broke at but in 0.89.3 it produces This is LuaTeX, Version beta-0.89.3 (TeX Live 2016) restricted system commands enabled. (./sp.tex Overfull \vbox (15749.96411pt too high) has occurred while \output is active and the brackets expand vertically for more than the page height. David
