FWIW, we're testing indexing on hadoop too, using LocalJobRunner, no MiniDFCluster (since we're not after testing HDFS, only our MapReduce logic) and it works on Windows and Linux.
Not everything is testable that way though, for instance there were bugs that were only exposed on a true HDFS (but not GPFS), but since we develop both on Windows and Linux, we chose not to try force MiniDFSCluster to run on Windows. I tried w/ the community, but it seemed they didn't care much about it. I don't know what Solr tests, and why it needs MiniDFSCluster, just pointing out that you can test MR jobs on Windows too. Shai On Tue, Jul 9, 2013 at 10:27 AM, Uwe Schindler <[email protected]> wrote: > Hi Mark, > > > > I am sure, it's horrible slowdoop! > > > > It doesn't bother me that you say slowdoop or say that you said that > hadoop > > was made by first year students on twitter - but I will point out it > kind of > > makes you look like a dick - a community of people work on that project > that > > is very similar to the one that works on Lucene/Solr. And I know you, so > it > > doesn't really affect my view of you, but it's kind of unbecoming for a > PMC > > chair to besmirch another Apache project by referring to it with a > derogatory > > nickname.... FWIW - no real skin off my nose - besmirch away if that's > what > > you enjoy. > > My problem is not with Hadoop, my problem is that this is part of main > Solr module and is not in a contrib module (where it should live). It was > committed without discussing about all the integration problems, like > test-only JAR files in different versions like we currently have. I have > nothing against Hadoop, but I feel like we should not use code that has a > bad design and is not platform independent (I am talking about the test > dependencies only!). Isn't it possible to test our code without using this > MiniDFSCluster? This component also seems to be the cause of the permgen > errors: We need to tune it or replace it by something that works correctly > - also on Windows! > > I agree the "slow" nickname is bad - the parts of hadoop code I looked at > should be updated, including the Cygwin requirement (MiniDFSCluster, UNIX > Shell code). But the response time to new issues seems to be low (see the > Locale problems where you submitted a 2-line patch), so I don't want to > submit patches or issues at the moment. > > Uwe > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
