For the font-cache, see https://github.com/matplotlib/matplotlib/issues/5836 ...there is some ~/.cache/matplotlib directory that you can delete to get rid of this.
To tell PyPlot to use the Conda Python, you need to set the PYTHON environment variable and re-build PyCall: ENV["PYTHON"]="" Pkg.build("PyCall")