Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
Mark Murphy: > On Mon, Mar 28, 2016, at 07:42, Hans-Christoph Steiner wrote: >>> To help prevent memory leaks, in case clients fail to unregister their >>> callbacks. >> >> I'd prefer to error/warn the user about this rather than reducing the >> impact of doing the wrong thing. I vaguely remember seeing such >> warnings in logcat. > > You have *far* greater faith in software developers than I do. I think I might have less, since I'd like to make it a hard fail error to force people to do the right thing :) >> I'm saying remove the keystore references from your code entirely, for >> now at least. > > OK, will do. With luck, I will submit pull requests later this week. Looking forward to it. .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex=0xE9E28DEA00AA5556 ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
On Mon, Mar 28, 2016, at 07:42, Hans-Christoph Steiner wrote: > > To help prevent memory leaks, in case clients fail to unregister their > > callbacks. > > I'd prefer to error/warn the user about this rather than reducing the > impact of doing the wrong thing. I vaguely remember seeing such > warnings in logcat. You have *far* greater faith in software developers than I do. > I'm saying remove the keystore references from your code entirely, for > now at least. OK, will do. With luck, I will submit pull requests later this week. -- Mark Murphy (a Commons Guy) https://commonsware.com | https://github.com/commonsguy https://commonsware.com/blog | https://twitter.com/commonsguy ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
Mark Murphy: > On Fri, Mar 25, 2016, at 05:36, Hans-Christoph Steiner wrote: >> * why did you use the custom WeakSet class rather than just a plain Set? > > To help prevent memory leaks, in case clients fail to unregister their > callbacks. I'd prefer to error/warn the user about this rather than reducing the impact of doing the wrong thing. I vaguely remember seeing such warnings in logcat. >> * keep things as a JAR as much as possible, not an AAR, i.e. by avoiding >> string resources, etc. I find it makes the library a lot easier to >> deploy in a variety of situations (ant!). > > What do you want to do about the keystore? > > In my sample, I moved that to an asset, as I was having problems > referencing the raw resource across library module boundaries. However, > if you're aiming to change packaging to be a JAR instead of your current > AAR, then the keystore cannot be either a raw resource or an asset. > > Options include: > > - Remove the keystore references from my code entirely, with an eye > towards revamping all this (e.g., your proposed "make it look like > Android N" approach). > > - Leave the keystore references more or less as-is, and tell users of > the library that they need to put the keystore in the proper spot in > assets/ (i.e., the old SQLCipher approach, before they switched to the > AAR). > > - Add something (to OrbotHelper?) that exposes the keystore from raw > resources via a Java API, then have my code use that API, and leave the > raw resource where it is, requiring AAR packaging. > > - Attempt to switch this to an old-style Java JAR resource (note: not > recommended, as I worry that the Android tools team will eventually > break that). > > And there may be other options that I'm not thinking of, as I just woke > up. I'm saying remove the keystore references from your code entirely, for now at least. I think that approach could be good, but I'd like to have it wrapped in a proper test suite before deploying it any more. Right now, the focus of this round of work is improved proxy/tor integration. .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex=0xE9E28DEA00AA5556 ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
On Fri, Mar 25, 2016, at 05:36, Hans-Christoph Steiner wrote: > * why did you use the custom WeakSet class rather than just a plain Set? To help prevent memory leaks, in case clients fail to unregister their callbacks. > * keep things as a JAR as much as possible, not an AAR, i.e. by avoiding > string resources, etc. I find it makes the library a lot easier to > deploy in a variety of situations (ant!). What do you want to do about the keystore? In my sample, I moved that to an asset, as I was having problems referencing the raw resource across library module boundaries. However, if you're aiming to change packaging to be a JAR instead of your current AAR, then the keystore cannot be either a raw resource or an asset. Options include: - Remove the keystore references from my code entirely, with an eye towards revamping all this (e.g., your proposed "make it look like Android N" approach). - Leave the keystore references more or less as-is, and tell users of the library that they need to put the keystore in the proper spot in assets/ (i.e., the old SQLCipher approach, before they switched to the AAR). - Add something (to OrbotHelper?) that exposes the keystore from raw resources via a Java API, then have my code use that API, and leave the raw resource where it is, requiring AAR packaging. - Attempt to switch this to an old-style Java JAR resource (note: not recommended, as I worry that the Android tools team will eventually break that). And there may be other options that I'm not thinking of, as I just woke up. > If you want to do it via pull requests, I'd do it one per HTTP API, and > another for the Orbot stuff. I'm not aware of other options for getting my stuff into your repo, other than pull requests, and possibly butterflies. https://xkcd.com/378/ Thanks! -- Mark Murphy (a Commons Guy) https://commonsware.com | https://github.com/commonsguy https://commonsware.com/blog | https://twitter.com/commonsguy ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
Mark Murphy: > On Mon, Mar 21, 2016, at 09:39, Hans-Christoph Steiner wrote: >> As for a plan for integrating this into NetCipher, I'd like to start >> with the HttpURLConnection first, especially the SNI stuff. Ideally, it >> would all fit into the existing TlsOnlySocket* stuff, without the need >> for a Sni-specific subclass. > > I am not completely clear where the rest of that thread left us. I am > assuming that you want to go ahead with the HttpURLConnection code per > your statement above. > > If so: > > - Just to confirm, you want this code to go into libnetcipher? yes! > - Is the package name that I chose (info.guardianproject.netcipher.hurl) > OK, or do you want that changed? As far as I understand it, the HttpURLConnection-specific stuff will just be folded into TlsOnlySocket*, so that will stay where it is. Then the other stuff is more general, like StrongBuilderBase.java, so it could go into i.g.n.client, with the Orbot stuff going into i.g.n.proxy. > - Is the API that I am using basically acceptable to you, or do you want > a major overhaul? Minor changes we can do after getting the code in your > repo, but if you're looking for what might involve a top-to-bottom > rewrite, I'd prefer to tackle that separately. I think things are ready to be merged with the existing general API. I might end up changing things a bit to put the LocalBroadcastManager style on equal footing as the Interfaces and Callbacks API. But I'll first have to figure out how to do that well. * why did you use the custom WeakSet class rather than just a plain Set? * keep things as a JAR as much as possible, not an AAR, i.e. by avoiding string resources, etc. I find it makes the library a lot easier to deploy in a variety of situations (ant!). > - What sort of pull request granularity do you want? Fewer larger ones > or many smaller ones? I'm a pull request n00b, so I apologize if the > answer to that question should be obvious. If you want to do it via pull requests, I'd do it one per HTTP API, and another for the Orbot stuff. > - Do you want me to open a tracking issue for this work anywhere (and if > so, where)? We've been discussing here already, so we can just stay on this list. > - Is there anything else that I should know before working on this and > lobbing pull requests over to you? I think we're ready to go! .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex=0xE9E28DEA00AA5556 ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
Mark Murphy: > On Fri, Feb 26, 2016, at 08:43, Hans-Christoph Steiner wrote: >> I tried a bit using setComponent() and a freshly installed and not >> started Orbot. That did not seem to start it either. > > I have reconfirmed that it is working for me. I had some bugs related to > detecting the installation, but the explicit Intent is sufficient to > take a freshly-installed Orbot (at least the edition on the Play Store) > and get it running to the point where we can make requests through the > HTTP proxy. Is this in the HTTPStacks.zip bundle already? I'd like to try it. I want to get this change and the SNI support into NetCipher ASAP to make a new release. .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex=0xE9E28DEA00AA5556 ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
On Fri, Feb 26, 2016, at 08:43, Hans-Christoph Steiner wrote: > I tried a bit using setComponent() and a freshly installed and not > started Orbot. That did not seem to start it either. Does the "allow other apps to start Orbot" setting come pre-checked? If not, that may be the source of the difficulty. -- Mark Murphy (a Commons Guy) https://commonsware.com | https://github.com/commonsguy https://commonsware.com/blog | https://twitter.com/commonsguy ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
Mark Murphy: > On Thu, Feb 25, 2016, at 08:40, Hans-Christoph Steiner wrote: >> So I guess the key bit there in terms of making the Intent start Orbot >> not matter what its state is the Intent.setComponent() call? > > Sorry, you lost me there. Is Intent.setComponent() the thing that makes the Orbot ACTION_START Intent launch Orbot even if it is in the stopped state after just being installed? >> I could make sense to make NetCipher work well >> with AndroidPinning, but even better would be to have a full >> pinning/TOFU support built-in. > > The choice of GPLv3 for a library, while honorable, won't be popular > among Android developers that pay attention to licenses. Yup, but that library is also not so complicated and limited in what it can do, so it would be nice to have a replacement. >> Here's the archtecture for that that >> I've been working on over the years: >> >> https://dev.guardianproject.info/projects/bazaar/wiki/Chained_TLS_Cert_Verification > > In a related area, I have TrustManagerBuilder in CWAC-Security: > > https://github.com/commonsguy/cwac-security/blob/master/TrustManagerBuilder.markdown Seems like we should join forces on this one! .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex=0xE9E28DEA00AA5556 ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
Mark Murphy: > On Thu, Feb 25, 2016, at 08:01, Hans-Christoph Steiner wrote: >> Any chance of viewing this in git? > > Not until it gets released. At that point, it will be in my book's > massive repo (https://github.com/commonsguy/cw-omnibus), along with the > rest of my book samples. > >> I'll start digging into these. I looked at netcipher-hurl, that seems >> to be a simplified duplication of what's already in the >> NetCipher.getHttpURLConnection() methods, based on the >> TlsOnlySocketFactory.java class. Was that deliberate? > > No. > > My biggest problem with all of this is that NetCipher is undocumented > AFAICT. Hence, I worked off of the sample app to determine what is and > is not your supported API. I didn't notice a getHttpURLConnection() > method. I'll run a comparison and make changes if needed -- thanks! The sample app is mostly for the ch.boye methods at this point, but yeah, we need to improve the docs and sample app. The HttpURLConnection stuff is quite simple, here are some examples: https://github.com/theScrabi/NewPipe/pull/136/files >>> I do not believe that setPackage() is sufficient, for the Orbot status >>> Intent, to get Orbot to run after a fresh installation, if the user has not >>> run the Orbot activity. As of Android 3.1, apps are installed in a >>> so-called “stopped state”, where manifest-registered receivers will not >>> receive broadcasts of implict Intents. Using an explicit Intent, as I am, >>> will help here. >> >> Ok, that's interesting. I couldn't find where you are making an >> explicit Intent for this. > > That's inside validateBroadcastIntent(), over in my CWAC-Security > library. For the purposes of my book, having it there is fine. For any > long-term NetCipher reuse, we'd want to copy the logic from there or do > something equivalent, so there's no dependency on something tiny like > this: > > https://github.com/commonsguy/cwac-security/blob/master/security/src/main/java/com/commonsware/cwac/security/SignatureUtils.java#L70-L152 This is like our "Trusted Intents" library, which reminds me that I've been meaning to discuss your versions of this stuff. :) FYI, ours is here: https://github.com/guardianproject/TrustedIntents So I guess the key bit there in terms of making the Intent start Orbot not matter what its state is the Intent.setComponent() call? >> It could easily take Orbot more than 30 seconds to complete the whole >> Tor startup sequence. That's a key reason for the Intent broadcasts >> with the OFF/STARTING/ON/STOPPING status. That let's the app then >> respond accordingly, like holding network connections when in the >> STARTING state. > > StatusCallback could be expanded for the other states. Again, it's not > obvious what is and is not part of the supported API. > > Expanding StatusCallback is out of scope for the current book sample > (like your sample, it doesn't really need it), but it's clearly > something that could be added as part of any long-term NetCipher reuse. I think I'm missing something. Orbot sends OFF/STARTING/ON/STOPPING status now. I don't think we need to extend the status states or change that stuff at all. It just needs to be clear to the app developer that if the app is using Tor and Orbot is not in the ON state, the app will not have working network connectivity. >> I think we should skip that approach entirely, and leave only where it >> was used for backwards compatibility. AndroidPinning provides a more >> manageable approach to certificate pinning. If you are interested in >> looking more into that, it could be a new chunk of work related to this. > > If you mean https://github.com/moxie0/AndroidPinning, that's GPLv3. > > Thanks! Yeah, that's the one. I could make sense to make NetCipher work well with AndroidPinning, but even better would be to have a full pinning/TOFU support built-in. Here's the archtecture for that that I've been working on over the years: https://dev.guardianproject.info/projects/bazaar/wiki/Chained_TLS_Cert_Verification .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex=0xE9E28DEA00AA5556 ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
On Thu, Feb 25, 2016, at 08:01, Hans-Christoph Steiner wrote: > Any chance of viewing this in git? Not until it gets released. At that point, it will be in my book's massive repo (https://github.com/commonsguy/cw-omnibus), along with the rest of my book samples. > I'll start digging into these. I looked at netcipher-hurl, that seems > to be a simplified duplication of what's already in the > NetCipher.getHttpURLConnection() methods, based on the > TlsOnlySocketFactory.java class. Was that deliberate? No. My biggest problem with all of this is that NetCipher is undocumented AFAICT. Hence, I worked off of the sample app to determine what is and is not your supported API. I didn't notice a getHttpURLConnection() method. I'll run a comparison and make changes if needed -- thanks! > > I do not believe that setPackage() is sufficient, for the Orbot status > > Intent, to get Orbot to run after a fresh installation, if the user has not > > run the Orbot activity. As of Android 3.1, apps are installed in a > > so-called “stopped state”, where manifest-registered receivers will not > > receive broadcasts of implict Intents. Using an explicit Intent, as I am, > > will help here. > > Ok, that's interesting. I couldn't find where you are making an > explicit Intent for this. That's inside validateBroadcastIntent(), over in my CWAC-Security library. For the purposes of my book, having it there is fine. For any long-term NetCipher reuse, we'd want to copy the logic from there or do something equivalent, so there's no dependency on something tiny like this: https://github.com/commonsguy/cwac-security/blob/master/security/src/main/java/com/commonsware/cwac/security/SignatureUtils.java#L70-L152 > It could easily take Orbot more than 30 seconds to complete the whole > Tor startup sequence. That's a key reason for the Intent broadcasts > with the OFF/STARTING/ON/STOPPING status. That let's the app then > respond accordingly, like holding network connections when in the > STARTING state. StatusCallback could be expanded for the other states. Again, it's not obvious what is and is not part of the supported API. Expanding StatusCallback is out of scope for the current book sample (like your sample, it doesn't really need it), but it's clearly something that could be added as part of any long-term NetCipher reuse. > I think we should skip that approach entirely, and leave only where it > was used for backwards compatibility. AndroidPinning provides a more > manageable approach to certificate pinning. If you are interested in > looking more into that, it could be a new chunk of work related to this. If you mean https://github.com/moxie0/AndroidPinning, that's GPLv3. Thanks! -- Mark Murphy (a Commons Guy) https://commonsware.com | https://github.com/commonsguy https://commonsware.com/blog | https://twitter.com/commonsguy ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
Re: [guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
Mark Murphy: > I have written a chapter for _The Busy Coder's Guide to Android > Development_ on the use of NetCipher. Along the way, I wrote integration > code to tie NetCipher into: > > - OkHttp3 (including using it with Retrofit) > - HttpURLConnection > - Apache's independent packaging of HttpClient > - Volley > > Here is the draft chapter, in PDF form: > > https://commonsware.com/misc/NetCipher.pdf > > One section is missing towards the end (you'll see a "TBD" there), but > it is otherwise complete. Forgive the sidebars, but since this is a new > chapter, it gets those on the first book update containing the chapter. > > Here is a ZIP file containing the sample project and HTTP integration > code: > > https://commonsware.com/misc/HTTPStacks.zip Any chance of viewing this in git? > When I ship the book update containing this chapter in the second half > of March, I'll release a copy of this chapter under a Creative Commons > license. Also, the HTTP integration code is all Apache License 2.0; you > are welcome to any of that if you want it. > > I welcome any feedback on the prose or the code! I'll start digging into these. I looked at netcipher-hurl, that seems to be a simplified duplication of what's already in the NetCipher.getHttpURLConnection() methods, based on the TlsOnlySocketFactory.java class. Was that deliberate? > I do not believe that setPackage() is sufficient, for the Orbot status > Intent, to get Orbot to run after a fresh installation, if the user has not > run the Orbot activity. As of Android 3.1, apps are installed in a so-called > “stopped state”, where manifest-registered receivers will not receive > broadcasts of implict Intents. Using an explicit Intent, as I am, will help > here. Ok, that's interesting. I couldn't find where you are making an explicit Intent for this. > While NetCipher works fine if Orbot is running, the Orbot auto-start based on > that broadcast > did not seem to work all that well, particularly on older/slower hardware. It > may be that my > 30-second timeout before assuming that we’ll never hear back from Orbot is > too short. It could easily take Orbot more than 30 seconds to complete the whole Tor startup sequence. That's a key reason for the Intent broadcasts with the OFF/STARTING/ON/STOPPING status. That let's the app then respond accordingly, like holding network connections when in the STARTING state. > The OrbotInitializer class that implements the callback mechanism, the > signature checking, > and so on, could in theory be merged into OrbotHelper. I specifically wanted > to avoid > changing anything in libnetcipher itself, which is why I broke it out into a > separate class. The end goal is to have this stuff all nicely integrated into its most natural homes: either the netcipher library or the upstream HTTP libraries. > • I had tons of trouble trying to use the Debian keystore you have as a raw > resource. The > build tools do not seem to want to let me reference that via an R.raw > identifier. To work > around that, I made a copy of that keystore in assets/, which is more > flexible. All else being > equal, I'd recommend that NetCipher go the assets/ route, but it may be > possible that the > R.raw problem can be overcome once some of this code gets merged directly into > libnetcipher. I think we should skip that approach entirely, and leave only where it was used for backwards compatibility. AndroidPinning provides a more manageable approach to certificate pinning. If you are interested in looking more into that, it could be a new chunk of work related to this. .hc -- PGP fingerprint: EE66 20C7 136B 0D2C 456C 0A4D E9E2 8DEA 00AA 5556 https://pgp.mit.edu/pks/lookup?op=vindex=0xE9E28DEA00AA5556 ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
[guardian-dev] NetCipher Book Chapter, and HTTP Stack Integration
I have written a chapter for _The Busy Coder's Guide to Android Development_ on the use of NetCipher. Along the way, I wrote integration code to tie NetCipher into: - OkHttp3 (including using it with Retrofit) - HttpURLConnection - Apache's independent packaging of HttpClient - Volley Here is the draft chapter, in PDF form: https://commonsware.com/misc/NetCipher.pdf One section is missing towards the end (you'll see a "TBD" there), but it is otherwise complete. Forgive the sidebars, but since this is a new chapter, it gets those on the first book update containing the chapter. Here is a ZIP file containing the sample project and HTTP integration code: https://commonsware.com/misc/HTTPStacks.zip When I ship the book update containing this chapter in the second half of March, I'll release a copy of this chapter under a Creative Commons license. Also, the HTTP integration code is all Apache License 2.0; you are welcome to any of that if you want it. I welcome any feedback on the prose or the code! -- Mark Murphy (a Commons Guy) https://commonsware.com | https://github.com/commonsguy https://commonsware.com/blog | https://twitter.com/commonsguy ___ List info: https://lists.mayfirst.org/mailman/listinfo/guardian-dev To unsubscribe, email: guardian-dev-unsubscr...@lists.mayfirst.org
