Unfortunately, the copy of Qt bundled with PhantomJS contains substantial modifications. Building PhantomJS with a system-provided stock Qt may *appear* to work, but will break a bunch of stuff that its users depend upon. See https://github.com/ariya/phantomjs/issues/13727#issuecomment-155693418 for more detail.
As a representative of upstream, we understand Debian's concerns about this (we are painfully aware of the difficulty of trying to keep up with Qt and Webkit) but we think it's probably best, right now, if Debian does not attempt to package PhantomJS at all. zw

