Hey David, As awful as it is, is shading the dep an option here?
J On Mon, Nov 11, 2019 at 11:05 AM David Ortiz <[email protected]> wrote: > Hey, > > We were doing some testing using EMR and found that Crunch has an > issue with Hadoop 2.8.5. Specifically, it comes from the changes made for > CRUNCH-679. It appears that the CrunchRenameCopyListing class was updated > to use the “shouldCopy” method of SimpleFileListing. > > if (!shouldCopy(fileStatus.getPath(), options)) { > return; > } > > Looking at the history of the SimpleFileListing class in hadoop-distcp, it > appears that from version 2.8.0 on, it no longer takes DistCp options as a > parameter, which causes jobs using CrunchRenameCopyListing on versions of > Hadoop at that point on to fail with a NoSuchMethodError. I’ve attempted > working around it by pulling the older hadoop-distcp package in using the > “packages” argument with Spark, but it does not seem to have helped. Any > other suggestions to try? > > Thanks, > Dave >
