Thanks for bringing this onto the mailing list, Jake. First, I should preface by saying that I created the ticket with the goal of initiating a discussion around Thrift's architecture going forward. There is no urgent or immediate concern (conversely, the nature of the ticket implies that changes, if any, are unlikely to be quick and would require serious thought/discussion).
The TL;DR version is this: I think the current Thrift repo is increasingly harder to maintain and test, especially when it comes to consistency of code quality and functionality across languages. Yes, I can use configure to specify exactly which libraries I want to build. But consider: * Say I'm a new Thrift user. I check out the repo and now I just want to quickly build Thrift and run some tests. Let's also assume that configure does a good job of choosing sensible defaults (which itself is suspect). What's the next step? Will 'make' in the top-dir build all the configured libraries? Will 'make check' (or similar) run all the tests? Last I tried, that didn't work. Even if it does, as more languages get added, it requires more and more plumbing to just keep all this together with make. * Every language has it's own ecosystem and way of doing things. Thrift should embrace them where possible, rather than try to shoe-horn everything into a single mold -- it leads to suboptimal use of the developers' time. For instance, Java works great with Maven and Scala with sbt. There's no clean way to integrate them both into a single top-level project using 'make' or whatever else, though. * This tight coupling between the IDL and the language libraries also makes it harder to experiment. Say I wanted to write a new Java library for Thrift. I'd end up a lot of time just trying to integrate this new library with the various build scripts and such. If I had compiler and a clean API to work with, I could just create a separate Maven project, drop in the compiler and voila! Conversely, say I wanted to write a new compiler in Python or something and test it against one of the existing libraries. Again, the simpler the inter-dependencies, the easier these things become. * It's unlikely that all the Thrift libraries will have the same level of code quality or even consistency in implementation. Right now there's no good mechanism for the devs to focus on a few, popular libraries in the core distribution, while still allowing other libraries to thrive in the ecosystem. A quick search in Jira will reveal that even among Java and C++, there's a host of subtle differences. All these issues will get worse over time. With the decoupling, each component can independently strengthen it's own tests: the core IDL compiler, the Java library, C++ library etc -- each using the best tool for the job. IMO it is a lot simpler to then put together an integration harness that can mix and match various client/server implementations to auto-generate a compatibility matrix. Finally, code has a tendency to linger. The older and larger the codebase, the harder it gets to streamline and modularize it. What I'm suggesting is not unlike the principles you would apply to any other large piece of code. Diwaker On Mon, Dec 12, 2011 at 5:56 PM, Jake Farrell <[email protected]> wrote: > Diwaker > What is the issue you are trying to address that you are seeing in your > day to day development with this ticket? Initially you could remove given > languages not in use from the compiler and this was removed in favor of > always having available every language that is possibly to be generated. > This should have no impact on any test cases or libraries you wish to not > include. Configure also offers the ability to compile without any of the > client libs with the --without-<LIB> option. This should allow you to > compile and test the client libs that you are looking at using and you will > not have to worry about dependencies for libraries you are not going to > use. What problem are you running into that you are trying to address? > > - Jake > > > > > On Dec 12, 2011, at 6:39 PM, Diwaker Gupta (Created) (JIRA) wrote: > > > Decouple Thrift IDL from generators > > ----------------------------------- > > > > Key: THRIFT-1463 > > URL: https://issues.apache.org/jira/browse/THRIFT-1463 > > Project: Thrift > > Issue Type: Wish > > Components: Compiler (General) > > Reporter: Diwaker Gupta > > Assignee: Jake Farrell > > > > > > While Thrift's broad language support is fantastic, it does impose many > constraints on day to day development. > > > > * The current design of the compiler make it hard to improve and test > the generator/library for a particular language. The codebase is monolithic > and hard to navigate. > > * Each language has it's own idiosyncrasies. For instance, the Java > library required ant to build, the Ruby library has other dependencies. > Running unit tests is slightly different for each language library. > Currently, all of this is duck-taped together (barely) using 'make', which > has its own flaws. > > * Adding support for a new language is fairly easy, but rather than > making the code more modular, it adds to the current complexity of the > codebase. For example, setting up Jenkins jobs to test/verify builds for a > new language take a while to come up to parity with other languages. > > > > I think Google's Protocol Buffer approach is instructive here. They're > trying to strip down the core compiler and decouple the IDL from language > specific stubs. For a rich environment like Thrift, I think this decoupling > is crucial to allow for a more maintainable and testable code base going > forward. To refresh: > > > > * the core compiler takes in a Thrift grammar file and generates an > intermediate representation: think of an in-memory AST > > * each supported language will be implemented via plugins, that can be > loaded at runtime by the compiler > > * the plugins take this AST and transform it into source code > > * each plugin can be in its own repository (consuming the compiler via a > git submodule, for instance). Plugins can freely choose their own build > system, unit test frameworks etc > > * a meta repository can contain the compiler, and the 'blessed' > (officially supported) languages. This meta repository can include > integration tests that will test language interoperability > > > > I realize, of course, that this will be a big change. But we have to > start somewhere. > > > > -- > > This message is automatically generated by JIRA. > > If you think it was sent incorrectly, please contact your JIRA > administrators: > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > > For more information on JIRA, see: > http://www.atlassian.com/software/jira > > > > > > -- http://maginatics.com
