If you're starting a new project and it happens to be in ANSI C, any functions declared as "static" are only visible to that source file. I suspect that isn't applicable here, but since lua is small, maybe adding "static" declarations to the functions-to-hide will work.
Another possibility is to tread the functions to hide as a separate shared lib, and treat it as a plugin. Have the main library load the hidden one, and then expose whatever needs to be exposed. Not sure exactly how that would be implemented, or if it even makes sense.