The latency is primarily due to the speed of light. The geosyncronous satllites hosting the service are 35,000 km away (roughly). So to send a packet one way from your house to the ISP's routers on the ground is a distance of around 70,000 km.
At 3.0x10^6 m/s that takes the packet 0.25 seconds, or 250ms just to get to the gateway into the internet. Add the time it takes to get to the destination server, lets say 100ms on the low end. That is 350ms for your packet to go one way, or 700ms both ways. With no network lag :) -- Brian
