Freedom of speech isn't an easy task: https://en.wikipedia.org/wiki/Freedom_of_speech#Limitations
Fortunately, calling people trolls and other names (ad-hominem indeed) and not backing it up is what you often did these days, so I can't say I would defend that.
But I can see why you're bothered by the upvote/downvote system without a proper reply though, because it's essentially the same problem. Minus the fact that the majority of downvotes (even without replies) are justified.
But I can agree that it's a bad and lazy habit.
