I just mean the converse: if the music was already paused, you probably don't want to unpause it just to cancel the binary skipping.
OK, well why couldn't you have a long press of pause would cancel? I don't think that would conflict with anything would it? so a short press would pause or resume the audio, and a long press would cancel. Alternatively, perhaps a long press could be used to undo the last skip? What do you think?