My comments are inline... 2011/5/5 Bjørn Borud <[email protected]>
> 1) re-do the parser using Antlr and do the compiler in Java or Python. > a Python runtime is more common and probably the most portable > alternative, but [ ... ] Interestingly, I've had similar thoughts, but for different reasons. I'm working to get THRIFT-1115 completed which adds more dynamic (de)serialization to the python core libraries. I "discovered" that there is a partially complete python compiler already developed and stored in SVN (under /branches/python-compiler/compiler/py/src/parser.py and nearby code). With some TLC and elbow grease, the partial python compiler will be able to generate the minimal code needed for a "dynamic" compilation (using TBase from THRIFT-1115 for de/serialization). This won't help any of the statically typed languages, but the python thrift compiler code isn't that far off from being able to produce code for multiple target languages. If you're interested in it, the code is very workable already. (I'm not sure this is the best/most-up-to-date SVN source for that code, but it's present nonetheless.) My personal opinion is that we shouldn't need to generate code for any language with dynamic typing, but instead take the approach the protocol buffers provides- a more dynamic "descriptor" based method, like what I've done in THRIFT-1115 for python. But on the other hand, it can save some CPU to generate out that boilerplate (de)serialization code, so there's a strong argument for supporting both use-cases. Anyhow... > I can also offer some perspective on why we chose Thrift early on and > why we are now gradually drifting away from it and reverting to > REST/JSON APIs. > I have seen a similar "drift" towards more REST+JSON over HTTP APIs at my place of work, but only because the target "developer" is not actually someone with a software development background or education... The over-promotion of "webmasters" and "HTML programmers" tends to lead to interfaces that speak to the lowest common denominator. The issues and pain-points are very different in that space (REST+JSON/HTTP). I personally tend to view Thrift as a terrific "core server" protocol, and pretty much anything over HTTP as a decent " user(i.e. confuser) facing ``API'' ". In my opinion, it's more a problem with the workforce lacking education and expertise than any technological shortcomings of Thrift. I don't want to start a flamewar, but the fact is that JSON is the "new XML" and is frequently treated AS THOUGH it is a "self-describing protocol". I don't want to preach to the choir, but the advantages of Thrift over JSON are numerous and proven, at least in the context of working with skilled software developers. (How many times have these so-called "REST APIs" changed their JSON output in backwards-incompatible ways, I cannot count... Or worse, some REST+JSON implementations aren't even using a true JSON serializer, so you end up with invalid JSON because a "web programmer" didn't realize that quotes need escaping, and so on.) Having a binary representation, and a strongly typed definition that behaves well in the old client/new server use-case really works well. Thrift gives assurance about how an application will behave in the presence of new data. I like that. so we chose Thrift hoping that it we would be able to make it work > painlessly for us. but it didn't. because it was a hassle to use on > anything but Linux and it even behaved a bit differently on different > macs in the office (I have forgotten the root cause). > I don't have any experience trying to work with Thrift on Windows, but for years I've worked with developers on Mac OS X who have had no problems at all developing/using Thrift based code that interacted with other platforms. To the best of my knowledge, Thrift has never made a change that breaks backwards compatibility (since ~Dec 2007 when I learned of it). I'm not sure what your pain points were, but I can only guess they were with the availability of the compiler for different platforms? we couldn't get other people to adopt it. (the main challenge was > when we had some Windows "customers" and parts of the system were > using a newer Thrift version to get around the OOM-problem in the > framing code. we'd have to assist them in getting the thrift compiler > built on Windows and then make it work with Maven. with none of the > developers in my office having a Windows machine. and of course, you > couldn't get pre-built windows binaries for arbitrary SVN revisions > from anywhere) > I think it's possible you expected too much of your Windows-using customers, or expected Thrift code to be more production-ready than it was at the time (or is today). No doubt you are correct in implying that the Thrift development "process" was not very solid, at least until recently. Around the days of Thrift 0.2 in late 2007 or early 2008, to use Thrift was an adventure. If you were lucky, you could check out the trunk and it would work. If not, then you had to go back to a previous trunk "snapshot". I used to jokingly refer to this as "the Tuesday or Wednesday build?" with co-workers at the time. Recently though, everything has changed. With the addition of Hudson (a.k.a. the Jenkins buildbot), we now cash in on and bank on the unit tests that exist in the codebase. Knowing that, I've made it a priority that whenever I submit a patch for a new feature, that I add test cases to exercise that new feature (like the python compressed zlib transport I added, and the python SSL enhancements I provided (certificate checking/specifiation, etc)...) > and to be honest, we spent a fair share of work ensuring that all the > code across all projects, being in different states of development > with different versions of the thrift compiler would work on every > platform we used. it wasn't the painless development we were used to. > I suspect the root problem was that for a long time, Thrift didn't really have "versions"... It had SVN _revisions_ (snapshots)... I used to complain to my coworkers that we should have chosen ProtocolBuffers for our IPC because at least Google had version numbers on their code, and we could depend on that to know what we were deploying to production. That problem has been totally fixed now. And with the addition of the TCompactProtocol classes (which I fixed for the python libraries when I enhanced the unit tests), there's no longer a significant size difference between serialized Thrift and ProtoBuf messages. (and adding to all these problems was the fact that since Thrift > wasn't released into Maven central at that time, projects like > Cassandra were trailing behind current versions. this meant that > anything that needed to talk to Cassandra would either have to use the > same version of Thrift as Cassandra or [ ... ] > I don't mean to question your claim, but I'm surprised that the version of Thrift (libraries or compiler) would make a difference to a client for an application that speaks Thrift. The on-wire protocol hasn't changed (to the best of my knowledge) in any significant way. Though, I tend not to use the Thrift-supplied RPC implementations in my own work, so maybe this is more due to the RPC quirks than the Thrift (de)serialization code? I'm sure a lot of these problems have been fixed, but the awkwardness > of using Thrift in a large, heterogenous development environment sort > of killed it for us. And we really wanted to use it. (I have > probably forgotten a truckload of the problems we had initially by > now. A lot has happened with our projects in the last year). That's really unfortunate- I'm sorry to hear it. I don't know why, but I've had exactly the opposite experience. In the past 3.5 years, Thrift has helped provide the glue that let me and my colleagues build a very successful distributed system, across three languages (Java, Python and Perl) that has scaled well and let us spend more time on cool application code, and less on wasting time writing more plumbing and wiring and valueless "copy message from format A to format B" type of junk. My single greatest criticism with the Thrift codebase is that the RPC implementations (services) are generally fragile (at least in Python and Perl). Since I started with Thrift so many years ago, that RPC implementations were still being developed for many target languages, I and my coworkers developed an RPC mechanism on top of Thrift that doesn't depend on the TProcessor, or TServer types. If I were to start anew, I would probably look at the java THsHaServer to begin from, but that's just personal bias. And honestly, since I've been using Thrift for over three years, I can't really complain about the RPC implementations. If it was a serious problem, I would contribute code to fix it. As it stands, Thrift lets you implement your own RPC if you want to, the same as ProtocolBuffers and JSON over HTTP. I hope this helps. Sharing my own experience, I don't mean to contradict anyone else's. I would love to hear more stories of how folks have used Thrift to solve their problems, and where it hasn't worked well. If anything matters, it's that we can all learn from others' experiences. I hope my story helps. -Will Pierce PS I don't work for Facebook, so no vested interest here.
