I have selected the same options as Cheng LIang: hadoop-2.4, hive, hive 0.12.0 . After a full Rebuild in IJ I still see the HiveShim errors.
I really do not know what is different. I had pulled three hours ago from github upstream master. Just for kicks i am trying PW's combination which uses 0.13.1 now.. But it appears there is something else going on here. Patrick/ Cheng: did you build on the command line using Maven first? I do that since in the past that had been required. 2014-10-28 21:57 GMT-07:00 Patrick Wendell <pwend...@gmail.com>: > I just started a totally fresh IntelliJ project importing from our > root pom. I used all the default options and I added "hadoop-2.4, > hive, hive-0.13.1" profiles. I was able to run spark core tests from > within IntelliJ. Didn't try anything beyond that, but FWIW this > worked. > > - Patrick > > On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <lian.cs....@gmail.com> wrote: > > You may first open the root pom.xml file in IDEA, and then go for menu > View > > / Tool Windows / Maven Projects, then choose desired Maven profile > > combination under the "Profiles" node (e.g. I usually use hadoop-2.4 + > hive > > + hive-0.12.0). IDEA will ask you to re-import the Maven projects, > confirm, > > then it should be OK. > > > > I can debug within IDEA with this approach. However, you have to clean > the > > whole project before debugging Spark within IDEA if you compiled the > project > > outside IDEA. Haven't got time to investigate this annoying issue. > > > > Also, you can remove sub projects unrelated to your tasks to accelerate > > compilation and/or avoid other IDEA build issues (e.g. Avro related Spark > > streaming build failure in IDEA). > > > > > > On 10/29/14 12:42 PM, Stephen Boesch wrote: > > > > I am interested specifically in how to build (and hopefully run/debug..) > > under Intellij. Your posts sound like command line maven - which has > always > > been working already. > > > > Do you have instructions for building in IJ? > > > > 2014-10-28 21:38 GMT-07:00 Cheng Lian <lian.cs....@gmail.com>: > >> > >> Yes, these two combinations work for me. > >> > >> > >> On 10/29/14 12:32 PM, Zhan Zhang wrote: > >>> > >>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable > >>> hive-0.12.0. Note that the thrift-server is not supported yet in > hive-0.13, > >>> but expected to go to upstream soon (Spark-3720). > >>> > >>> Thanks. > >>> > >>> Zhan Zhang > >>> > >>> > >>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <java...@gmail.com> > wrote: > >>> > >>>> Thanks Patrick for the heads up. > >>>> > >>>> I have not been successful to discover a combination of profiles (i.e. > >>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with > >>>> maven. Anyone who knows how to handle this - a quick note here would > be > >>>> appreciated. > >>>> > >>>> > >>>> > >>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pwend...@gmail.com>: > >>>> > >>>>> Hey Stephen, > >>>>> > >>>>> In some cases in the maven build we now have pluggable source > >>>>> directories based on profiles using the maven build helper plug-in. > >>>>> This is necessary to support cross building against different Hive > >>>>> versions, and there will be additional instances of this due to > >>>>> supporting scala 2.11 and 2.10. > >>>>> > >>>>> In these cases, you may need to add source locations explicitly to > >>>>> intellij if you want the entire project to compile there. > >>>>> > >>>>> Unfortunately as long as we support cross-building like this, it will > >>>>> be an issue. Intellij's maven support does not correctly detect our > >>>>> use of the maven-build-plugin to add source directories. > >>>>> > >>>>> We should come up with a good set of instructions on how to import > the > >>>>> pom files + add the few extra source directories. Off hand I am not > >>>>> sure exactly what the correct sequence is. > >>>>> > >>>>> - Patrick > >>>>> > >>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <java...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>> Hi Matei, > >>>>>> Until my latest pull from upstream/master it had not been > necessary > >>>>>> to > >>>>>> add the hive profile: is it now?? > >>>>>> > >>>>>> I am not using sbt gen-idea. The way to open in intellij has been to > >>>>>> Open > >>>>>> the parent directory. IJ recognizes it as a maven project. > >>>>>> > >>>>>> There are several steps to do surgery on the yarn-parent / yarn > >>>>>> projects > >>>>> > >>>>> , > >>>>>> > >>>>>> then do a full rebuild. That was working until one week ago. > >>>>>> Intellij/maven is presently broken in two ways: this hive shim > >>>>>> (which > >>>>> > >>>>> may > >>>>>> > >>>>>> yet hopefully be a small/simple fix - let us see) and (2) the > >>>>>> "NoClassDefFoundError > >>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a > >>>>> > >>>>> serious > >>>>>> > >>>>>> problem . > >>>>>> > >>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <matei.zaha...@gmail.com>: > >>>>>> > >>>>>>> Hi Stephen, > >>>>>>> > >>>>>>> How did you generate your Maven workspace? You need to make sure > the > >>>>> > >>>>> Hive > >>>>>>> > >>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea. > >>>>>>> > >>>>>>> Matei > >>>>>>> > >>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <java...@gmail.com> > >>>>> > >>>>> wrote: > >>>>>>>> > >>>>>>>> I have run on the command line via maven and it is fine: > >>>>>>>> > >>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn > >>>>> > >>>>> -Phadoop-2.3 > >>>>>>>> > >>>>>>>> compile package install > >>>>>>>> > >>>>>>>> > >>>>>>>> But with the latest code Intellij builds do not work. Following is > >>>>> > >>>>> one of > >>>>>>>> > >>>>>>>> 26 similar errors: > >>>>>>>> > >>>>>>>> > >>>>>>>> Error:(173, 38) not found: value HiveShim > >>>>>>>> > >>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize)) > >>>>>>>> > >>>>>>>> ^ > >>>>>>> > >>>>>>> > >>> > >> > > > > >
