You may first open the root pom.xml file in IDEA, and then go for menu
View / Tool Windows / Maven Projects, then choose desired Maven profile
combination under the "Profiles" node (e.g. I usually use hadoop-2.4 +
hive + hive-0.12.0). IDEA will ask you to re-import the Maven projects,
confirm, then it should be OK.
I can debug within IDEA with this approach. However, you have to clean
the whole project before debugging Spark within IDEA if you compiled the
project outside IDEA. Haven't got time to investigate this annoying issue.
Also, you can remove sub projects unrelated to your tasks to accelerate
compilation and/or avoid other IDEA build issues (e.g. Avro related
Spark streaming build failure in IDEA).
On 10/29/14 12:42 PM, Stephen Boesch wrote:
I am interested specifically in how to build (and hopefully
run/debug..) under Intellij. Your posts sound like command line maven
- which has always been working already.
Do you have instructions for building in IJ?
2014-10-28 21:38 GMT-07:00 Cheng Lian <lian.cs....@gmail.com
<mailto:lian.cs....@gmail.com>>:
Yes, these two combinations work for me.
On 10/29/14 12:32 PM, Zhan Zhang wrote:
-Phive is to enable hive-0.13.1 and "-Phive -Phive-0.12.0” is
to enable hive-0.12.0. Note that the thrift-server is not
supported yet in hive-0.13, but expected to go to upstream
soon (Spark-3720).
Thanks.
Zhan Zhang
On Oct 28, 2014, at 9:09 PM, Stephen Boesch
<java...@gmail.com <mailto:java...@gmail.com>> wrote:
Thanks Patrick for the heads up.
I have not been successful to discover a combination of
profiles (i.e.
enabling hive or hive-0.12.0 or hive-13.0) that works in
Intellij with
maven. Anyone who knows how to handle this - a quick note
here would be
appreciated.
2014-10-28 20:20 GMT-07:00 Patrick Wendell
<pwend...@gmail.com <mailto:pwend...@gmail.com>>:
Hey Stephen,
In some cases in the maven build we now have pluggable
source
directories based on profiles using the maven build
helper plug-in.
This is necessary to support cross building against
different Hive
versions, and there will be additional instances of
this due to
supporting scala 2.11 and 2.10.
In these cases, you may need to add source locations
explicitly to
intellij if you want the entire project to compile there.
Unfortunately as long as we support cross-building
like this, it will
be an issue. Intellij's maven support does not
correctly detect our
use of the maven-build-plugin to add source directories.
We should come up with a good set of instructions on
how to import the
pom files + add the few extra source directories. Off
hand I am not
sure exactly what the correct sequence is.
- Patrick
On Tue, Oct 28, 2014 at 7:57 PM, Stephen Boesch
<java...@gmail.com <mailto:java...@gmail.com>> wrote:
Hi Matei,
Until my latest pull from upstream/master it had
not been necessary to
add the hive profile: is it now??
I am not using sbt gen-idea. The way to open in
intellij has been to Open
the parent directory. IJ recognizes it as a maven
project.
There are several steps to do surgery on the
yarn-parent / yarn projects
,
then do a full rebuild. That was working until
one week ago.
Intellij/maven is presently broken in two ways:
this hive shim (which
may
yet hopefully be a small/simple fix - let us see)
and (2) the
"NoClassDefFoundError
on ThreadFactoryBuilder" from my prior emails -and
which is quite a
serious
problem .
2014-10-28 19:46 GMT-07:00 Matei Zaharia
<matei.zaha...@gmail.com
<mailto:matei.zaha...@gmail.com>>:
Hi Stephen,
How did you generate your Maven workspace? You
need to make sure the
Hive
profile is enabled for it. For example sbt/sbt
-Phive gen-idea.
Matei
On Oct 28, 2014, at 7:42 PM, Stephen
Boesch <java...@gmail.com
<mailto:java...@gmail.com>>
wrote:
I have run on the command line via maven
and it is fine:
mvn -Dscalastyle.failOnViolation=false
-DskipTests -Pyarn
-Phadoop-2.3
compile package install
But with the latest code Intellij builds
do not work. Following is
one of
26 similar errors:
Error:(173, 38) not found: value HiveShim
Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize))
^
