Cheng - to make it recognize the new HiveShim for 0.12 I had to click on spark-hive under "packages" in the left pane, then go to "Open Module Settings" - then explicitly add the v0.12.0/src/main/scala folder to the sources by navigating to it and then <ctrl>+click to add it as a source. Did you have to do this?
On Tue, Oct 28, 2014 at 9:57 PM, Patrick Wendell <pwend...@gmail.com> wrote: > I just started a totally fresh IntelliJ project importing from our > root pom. I used all the default options and I added "hadoop-2.4, > hive, hive-0.13.1" profiles. I was able to run spark core tests from > within IntelliJ. Didn't try anything beyond that, but FWIW this > worked. > > - Patrick > > On Tue, Oct 28, 2014 at 9:54 PM, Cheng Lian <lian.cs....@gmail.com> wrote: >> You may first open the root pom.xml file in IDEA, and then go for menu View >> / Tool Windows / Maven Projects, then choose desired Maven profile >> combination under the "Profiles" node (e.g. I usually use hadoop-2.4 + hive >> + hive-0.12.0). IDEA will ask you to re-import the Maven projects, confirm, >> then it should be OK. >> >> I can debug within IDEA with this approach. However, you have to clean the >> whole project before debugging Spark within IDEA if you compiled the project >> outside IDEA. Haven't got time to investigate this annoying issue. >> >> Also, you can remove sub projects unrelated to your tasks to accelerate >> compilation and/or avoid other IDEA build issues (e.g. Avro related Spark >> streaming build failure in IDEA). >> >> >> On 10/29/14 12:42 PM, Stephen Boesch wrote: >> >> I am interested specifically in how to build (and hopefully run/debug..) >> under Intellij. Your posts sound like command line maven - which has always >> been working already. >> >> Do you have instructions for building in IJ? >> >> 2014-10-28 21:38 GMT-07:00 Cheng Lian <lian.cs....@gmail.com>: >>> >>> Yes, these two combinations work for me. >>> >>> >>> On 10/29/14 12:32 PM, Zhan Zhang wrote: >>>> >>>> -Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0" is to enable >>>> hive-0.12.0. Note that the thrift-server is not supported yet in hive-0.13, >>>> but expected to go to upstream soon (Spark-3720). >>>> >>>> Thanks. >>>> >>>> Zhan Zhang >>>> >>>> >>>> On Oct 28, 2014, at 9:09 PM, Stephen Boesch <java...@gmail.com> wrote: >>>> >>>>> Thanks Patrick for the heads up. >>>>> >>>>> I have not been successful to discover a combination of profiles (i.e. >>>>> enabling hive or hive-0.12.0 or hive-13.0) that works in Intellij with >>>>> maven. Anyone who knows how to handle this - a quick note here would be >>>>> appreciated. >>>>> >>>>> >>>>> >>>>> 2014-10-28 20:20 GMT-07:00 Patrick Wendell <pwend...@gmail.com>: >>>>> >>>>>> Hey Stephen, >>>>>> >>>>>> In some cases in the maven build we now have pluggable source >>>>>> directories based on profiles using the maven build helper plug-in. >>>>>> This is necessary to support cross building against different Hive >>>>>> versions, and there will be additional instances of this due to >>>>>> supporting scala 2.11 and 2.10. >>>>>> >>>>>> In these cases, you may need to add source locations explicitly to >>>>>> intellij if you want the entire project to compile there. >>>>>> >>>>>> Unfortunately as long as we support cross-building like this, it will >>>>>> be an issue. Intellij's maven support does not correctly detect our >>>>>> use of the maven-build-plugin to add source directories. >>>>>> >>>>>> We should come up with a good set of instructions on how to import the >>>>>> pom files + add the few extra source directories. Off hand I am not >>>>>> sure exactly what the correct sequence is. >>>>>> >>>>>> - Patrick >>>>>> >>>>>> On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch <java...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>> Hi Matei, >>>>>>> Until my latest pull from upstream/master it had not been necessary >>>>>>> to >>>>>>> add the hive profile: is it now?? >>>>>>> >>>>>>> I am not using sbt gen-idea. The way to open in intellij has been to >>>>>>> Open >>>>>>> the parent directory. IJ recognizes it as a maven project. >>>>>>> >>>>>>> There are several steps to do surgery on the yarn-parent / yarn >>>>>>> projects >>>>>> >>>>>> , >>>>>>> >>>>>>> then do a full rebuild. That was working until one week ago. >>>>>>> Intellij/maven is presently broken in two ways: this hive shim >>>>>>> (which >>>>>> >>>>>> may >>>>>>> >>>>>>> yet hopefully be a small/simple fix - let us see) and (2) the >>>>>>> "NoClassDefFoundError >>>>>>> on ThreadFactoryBuilder" from my prior emails -and which is quite a >>>>>> >>>>>> serious >>>>>>> >>>>>>> problem . >>>>>>> >>>>>>> 2014-10-28 19:46 GMT-07:00 Matei Zaharia <matei.zaha...@gmail.com>: >>>>>>> >>>>>>>> Hi Stephen, >>>>>>>> >>>>>>>> How did you generate your Maven workspace? You need to make sure the >>>>>> >>>>>> Hive >>>>>>>> >>>>>>>> profile is enabled for it. For example sbt/sbt -Phive gen-idea. >>>>>>>> >>>>>>>> Matei >>>>>>>> >>>>>>>>> On Oct 28, 2014, at 7:42 PM, Stephen Boesch <java...@gmail.com> >>>>>> >>>>>> wrote: >>>>>>>>> >>>>>>>>> I have run on the command line via maven and it is fine: >>>>>>>>> >>>>>>>>> mvn -Dscalastyle.failOnViolation=false -DskipTests -Pyarn >>>>>> >>>>>> -Phadoop-2.3 >>>>>>>>> >>>>>>>>> compile package install >>>>>>>>> >>>>>>>>> >>>>>>>>> But with the latest code Intellij builds do not work. Following is >>>>>> >>>>>> one of >>>>>>>>> >>>>>>>>> 26 similar errors: >>>>>>>>> >>>>>>>>> >>>>>>>>> Error:(173, 38) not found: value HiveShim >>>>>>>>> >>>>>>>> Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize)) >>>>>>>>> >>>>>>>>> ^ >>>>>>>> >>>>>>>> >>>> >>> >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
