In /WEB-INF/flex/remoting-config.xml, you can either set the default channel to be just my-secure-cfamf, or explicity configure the channels on the <destination> in question.
If you set the RO channelSet property it will use that ChannelSet to connect rather than the configuration (in fact, you can stop specifying -services command line configuration argument if you manually create your own ChannelSets for RemoteObject based apps) - just make sure you only assign one channel in the channelset and then use this simplified scenario to debug what the problem is (rather than complicating it with failover URLs etc). One thing you may have to do is check if it's a problem only in MSIE. If it is, you may be running into the known issue with HTTPS responses that contain no-cache headers. You seem to have the correct <add-no-cache-headers> property set to false in your secure channel-definition, but that's not a guarantee that something else is not adding this header, such as another filter upstream or even your webserver. You should use a client side HTTP Sniffer such as Paros Proxy with MSIE which will allow you to see your HTTPS responses decrypted to check raw header information. Also watch out for known issues with MSIE and chunked encoded + gzip compressed responses over SSL as this will also fail. Finally, if you find that it's an issue with Firefox only... there is one obscure but known issue with this platform. If your SWF is hosted somewhere other than the CF server, then a crossdomain.xml is probably being requested first before your connection is made (due to sandbox security requirements of the Flash Player in the browser). If this request for crossdomain.xml is challenged for HTTP Basic credentials over an SSL connection that contains a port then it may just fail silently.