You might look at the geometry library in Fran (you can find a link in
haskell.org to Fran if you need it). It implements points and vectors
similar to what you're looking for. To answer your questions, you
can't (in standard Haskell) use the conventional operator names for
all of the operations. You can bring vectors into the Num class but
you can't get a type signature
(-) :: Point -> Point -> Vector
to fit into the standard Num class.
I can't really address performance issues, but you will definitely
need to use strictness annotations, probably in your data structures,
if you want C-like performance!
John