Hello Shan: Were almost have the same problem but mine is for a AOI handler with a robot arm which I need to pick (x,y) pairs along the work envelope. Anyway, for the most math part it will involve are coordinate transformation from a "mouse coordinates" to "real world coordinates" and your image processing textbook (I use McGrawHill Computer Graphics by Hill) will outline both the code and the math but LabVIEW will not have a facility to do with an actual VI instead you have to use the array manipulations and treat them coordinates as matrix elements. There is a Math VI and G-Math VI or a MatLAB call you can use for coordinate matrix manipulations as long as you have the math quite figured out in paper already.
