Newer BIND versions (9.4 and newer) have the "allow-query-cache" option which is used to specify which hosts are allowed to get answers from the cache. (I don't think you are seeing recursion, but just what is already in your cache.)
With older versions, a workaround is to have a default allow-query for
just your local networks (like your allow-recursion) in the options and
then open up allow-query { any; }; just within your specific zone
statements.
