Another thing to note is that QTime does not include the time it takes to retrieve the stored documents to include in the response. So if you're using a high rows value in your query, QTime may be much smaller than the actual time Solr spends generating the response.
Try adding rows=1 to your query and see if it still takes that long. -Michael