Re: Building Spark 2.X in Intellij
I just checked out completely fresh directory and created new IJ project. Then followed your tip for adding the avro source. Here is an additional set of errors Error:(31, 12) object razorvine is not a member of package net import net.razorvine.pickle.{IObjectPickler, Opcodes, Pickler} ^ Error:(779, 49) not found: type IObjectPickler class PythonMessageAndMetadataPickler extends IObjectPickler { ^ Error:(783, 7) not found: value Pickler Pickler.registerCustomPickler(classOf[PythonMessageAndMetadata], this) ^ Error:(784, 7) not found: value Pickler Pickler.registerCustomPickler(this.getClass, this) ^ Error:(787, 57) not found: type Pickler def pickle(obj: Object, out: OutputStream, pickler: Pickler) { ^ Error:(789, 19) not found: value Opcodes out.write(Opcodes.GLOBAL) ^ Error:(794, 19) not found: value Opcodes out.write(Opcodes.MARK) ^ Error:(800, 19) not found: value Opcodes out.write(Opcodes.TUPLE) ^ Error:(801, 19) not found: value Opcodes out.write(Opcodes.REDUCE) ^ 2016-06-22 23:49 GMT-07:00 Stephen Boesch : > Thanks Jeff - I remember that now from long time ago. After making that > change the next errors are: > > Error:scalac: missing or invalid dependency detected while loading class > file 'RDDOperationScope.class'. > Could not access term fasterxml in package com, > because it (or its dependencies) are missing. Check your build definition > for > missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see > the problematic classpath.) > A full rebuild may help if 'RDDOperationScope.class' was compiled against > an incompatible version of com. > Error:scalac: missing or invalid dependency detected while loading class > file 'RDDOperationScope.class'. > Could not access term jackson in value com.fasterxml, > because it (or its dependencies) are missing. Check your build definition > for > missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see > the problematic classpath.) > A full rebuild may help if 'RDDOperationScope.class' was compiled against > an incompatible version of com.fasterxml. > Error:scalac: missing or invalid dependency detected while loading class > file 'RDDOperationScope.class'. > Could not access term annotation in value com.jackson, > because it (or its dependencies) are missing. Check your build definition > for > missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see > the problematic classpath.) > A full rebuild may help if 'RDDOperationScope.class' was compiled against > an incompatible version of com.jackson. > Error:scalac: missing or invalid dependency detected while loading class > file 'RDDOperationScope.class'. > Could not access term JsonInclude in value com.annotation, > because it (or its dependencies) are missing. Check your build definition > for > missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see > the problematic classpath.) > A full rebuild may help if 'RDDOperationScope.class' was compiled against > an incompatible version of com.annotation. > Warning:scalac: Class org.jboss.netty.channel.ChannelFactory not found - > continuing with a stub. > > > 2016-06-22 23:39 GMT-07:00 Jeff Zhang : > >> You need to >> spark/external/flume-sink/target/scala-2.11/src_managed/main/compiled_avro >> under build path, this is the only thing you need to do manually if I >> remember correctly. >> >> >> >> On Thu, Jun 23, 2016 at 2:30 PM, Stephen Boesch >> wrote: >> >>> Hi Jeff, >>> I'd like to understand what may be different. I have rebuilt and >>> reimported many times. Just now I blew away the .idea/* and *.iml to start >>> from scratch. I just opened the $SPARK_HOME directory from intellij File | >>> Open . After it finished the initial import I tried to run one of the >>> Examples - and it fails in the build: >>> >>> Here are the errors I see: >>> >>> Error:(45, 66) not found: type SparkFlumeProtocol >>> val transactionTimeout: Int, val backOffInterval: Int) extends >>> SparkFlumeProtocol with Logging { >>> ^ >>> Error:(70, 39) not found: type EventBatch >>> override def getEventBatch(n: Int): EventBatch = { >>> ^ >>> Error:(85, 13) not found: type EventBatch >>> new EventBatch("Spark sink has been stopped!", "", >>> java.util.Collections.emptyList()) >>> ^ >>> >>> /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala >>> Error:(80, 22) not found: type EventBatch >>> def getEventBatch: EventBatch = { >>> ^ >>> Error:(48, 37) not found: type EventBatch >>> @volatile private var eventBatch: EventBatch = new EventBatch("Unknown >>> Error",
Re: Building Spark 2.X in Intellij
Thanks Jeff - I remember that now from long time ago. After making that change the next errors are: Error:scalac: missing or invalid dependency detected while loading class file 'RDDOperationScope.class'. Could not access term fasterxml in package com, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.) A full rebuild may help if 'RDDOperationScope.class' was compiled against an incompatible version of com. Error:scalac: missing or invalid dependency detected while loading class file 'RDDOperationScope.class'. Could not access term jackson in value com.fasterxml, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.) A full rebuild may help if 'RDDOperationScope.class' was compiled against an incompatible version of com.fasterxml. Error:scalac: missing or invalid dependency detected while loading class file 'RDDOperationScope.class'. Could not access term annotation in value com.jackson, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.) A full rebuild may help if 'RDDOperationScope.class' was compiled against an incompatible version of com.jackson. Error:scalac: missing or invalid dependency detected while loading class file 'RDDOperationScope.class'. Could not access term JsonInclude in value com.annotation, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.) A full rebuild may help if 'RDDOperationScope.class' was compiled against an incompatible version of com.annotation. Warning:scalac: Class org.jboss.netty.channel.ChannelFactory not found - continuing with a stub. 2016-06-22 23:39 GMT-07:00 Jeff Zhang : > You need to > spark/external/flume-sink/target/scala-2.11/src_managed/main/compiled_avro > under build path, this is the only thing you need to do manually if I > remember correctly. > > > > On Thu, Jun 23, 2016 at 2:30 PM, Stephen Boesch wrote: > >> Hi Jeff, >> I'd like to understand what may be different. I have rebuilt and >> reimported many times. Just now I blew away the .idea/* and *.iml to start >> from scratch. I just opened the $SPARK_HOME directory from intellij File | >> Open . After it finished the initial import I tried to run one of the >> Examples - and it fails in the build: >> >> Here are the errors I see: >> >> Error:(45, 66) not found: type SparkFlumeProtocol >> val transactionTimeout: Int, val backOffInterval: Int) extends >> SparkFlumeProtocol with Logging { >> ^ >> Error:(70, 39) not found: type EventBatch >> override def getEventBatch(n: Int): EventBatch = { >> ^ >> Error:(85, 13) not found: type EventBatch >> new EventBatch("Spark sink has been stopped!", "", >> java.util.Collections.emptyList()) >> ^ >> >> /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala >> Error:(80, 22) not found: type EventBatch >> def getEventBatch: EventBatch = { >> ^ >> Error:(48, 37) not found: type EventBatch >> @volatile private var eventBatch: EventBatch = new EventBatch("Unknown >> Error", "", >> ^ >> Error:(48, 54) not found: type EventBatch >> @volatile private var eventBatch: EventBatch = new EventBatch("Unknown >> Error", "", >> ^ >> Error:(115, 41) not found: type SparkSinkEvent >> val events = new util.ArrayList[SparkSinkEvent](maxBatchSize) >> ^ >> Error:(146, 28) not found: type EventBatch >> eventBatch = new EventBatch("", seqNum, events) >>^ >> >> /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSinkUtils.scala >> Error:(25, 27) not found: type EventBatch >> def isErrorBatch(batch: EventBatch): Boolean = { >> ^ >> >> /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSink.scala >> Error:(86, 51) not found: type SparkFlumeProtocol >> val responder = new SpecificResponder(classOf[SparkFlumeProtocol], >> handler.get) >> ^ >> >> >> Note: this is just the first batch of errors. >> >> >> >> >> 2016-06-22 20:50 GMT-07:00 Jeff Zhang : >> >>> It works well with me. You can try reimport it into intellij. >>> >>> On Thu, Jun 23, 2016 at 10:25 AM, Stephen Boesch >>> wrote: >>> Building inside intellij is an ever moving target. Anyo
Re: Building Spark 2.X in Intellij
You need to spark/external/flume-sink/target/scala-2.11/src_managed/main/compiled_avro under build path, this is the only thing you need to do manually if I remember correctly. On Thu, Jun 23, 2016 at 2:30 PM, Stephen Boesch wrote: > Hi Jeff, > I'd like to understand what may be different. I have rebuilt and > reimported many times. Just now I blew away the .idea/* and *.iml to start > from scratch. I just opened the $SPARK_HOME directory from intellij File | > Open . After it finished the initial import I tried to run one of the > Examples - and it fails in the build: > > Here are the errors I see: > > Error:(45, 66) not found: type SparkFlumeProtocol > val transactionTimeout: Int, val backOffInterval: Int) extends > SparkFlumeProtocol with Logging { > ^ > Error:(70, 39) not found: type EventBatch > override def getEventBatch(n: Int): EventBatch = { > ^ > Error:(85, 13) not found: type EventBatch > new EventBatch("Spark sink has been stopped!", "", > java.util.Collections.emptyList()) > ^ > > /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala > Error:(80, 22) not found: type EventBatch > def getEventBatch: EventBatch = { > ^ > Error:(48, 37) not found: type EventBatch > @volatile private var eventBatch: EventBatch = new EventBatch("Unknown > Error", "", > ^ > Error:(48, 54) not found: type EventBatch > @volatile private var eventBatch: EventBatch = new EventBatch("Unknown > Error", "", > ^ > Error:(115, 41) not found: type SparkSinkEvent > val events = new util.ArrayList[SparkSinkEvent](maxBatchSize) > ^ > Error:(146, 28) not found: type EventBatch > eventBatch = new EventBatch("", seqNum, events) >^ > > /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSinkUtils.scala > Error:(25, 27) not found: type EventBatch > def isErrorBatch(batch: EventBatch): Boolean = { > ^ > > /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSink.scala > Error:(86, 51) not found: type SparkFlumeProtocol > val responder = new SpecificResponder(classOf[SparkFlumeProtocol], > handler.get) > ^ > > > Note: this is just the first batch of errors. > > > > > 2016-06-22 20:50 GMT-07:00 Jeff Zhang : > >> It works well with me. You can try reimport it into intellij. >> >> On Thu, Jun 23, 2016 at 10:25 AM, Stephen Boesch >> wrote: >> >>> >>> Building inside intellij is an ever moving target. Anyone have the >>> magical procedures to get it going for 2.X? >>> >>> There are numerous library references that - although included in the >>> pom.xml build - are for some reason not found when processed within >>> Intellij. >>> >> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > -- Best Regards Jeff Zhang
Re: Building Spark 2.X in Intellij
Hi Jeff, I'd like to understand what may be different. I have rebuilt and reimported many times. Just now I blew away the .idea/* and *.iml to start from scratch. I just opened the $SPARK_HOME directory from intellij File | Open . After it finished the initial import I tried to run one of the Examples - and it fails in the build: Here are the errors I see: Error:(45, 66) not found: type SparkFlumeProtocol val transactionTimeout: Int, val backOffInterval: Int) extends SparkFlumeProtocol with Logging { ^ Error:(70, 39) not found: type EventBatch override def getEventBatch(n: Int): EventBatch = { ^ Error:(85, 13) not found: type EventBatch new EventBatch("Spark sink has been stopped!", "", java.util.Collections.emptyList()) ^ /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/TransactionProcessor.scala Error:(80, 22) not found: type EventBatch def getEventBatch: EventBatch = { ^ Error:(48, 37) not found: type EventBatch @volatile private var eventBatch: EventBatch = new EventBatch("Unknown Error", "", ^ Error:(48, 54) not found: type EventBatch @volatile private var eventBatch: EventBatch = new EventBatch("Unknown Error", "", ^ Error:(115, 41) not found: type SparkSinkEvent val events = new util.ArrayList[SparkSinkEvent](maxBatchSize) ^ Error:(146, 28) not found: type EventBatch eventBatch = new EventBatch("", seqNum, events) ^ /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSinkUtils.scala Error:(25, 27) not found: type EventBatch def isErrorBatch(batch: EventBatch): Boolean = { ^ /git/spark/external/flume-sink/src/main/scala/org/apache/spark/streaming/flume/sink/SparkSink.scala Error:(86, 51) not found: type SparkFlumeProtocol val responder = new SpecificResponder(classOf[SparkFlumeProtocol], handler.get) ^ Note: this is just the first batch of errors. 2016-06-22 20:50 GMT-07:00 Jeff Zhang : > It works well with me. You can try reimport it into intellij. > > On Thu, Jun 23, 2016 at 10:25 AM, Stephen Boesch > wrote: > >> >> Building inside intellij is an ever moving target. Anyone have the >> magical procedures to get it going for 2.X? >> >> There are numerous library references that - although included in the >> pom.xml build - are for some reason not found when processed within >> Intellij. >> > > > > -- > Best Regards > > Jeff Zhang >
Re: Building Spark 2.X in Intellij
I had some errors like SqlBaseParser class missing, and figured out I needed to get these classes from SqlBase.g4 using antlr4. It works fine now. On Thu, Jun 23, 2016 at 9:20 AM, Jeff Zhang wrote: > It works well with me. You can try reimport it into intellij. > > On Thu, Jun 23, 2016 at 10:25 AM, Stephen Boesch > wrote: > >> >> Building inside intellij is an ever moving target. Anyone have the >> magical procedures to get it going for 2.X? >> >> There are numerous library references that - although included in the >> pom.xml build - are for some reason not found when processed within >> Intellij. >> > > > > -- > Best Regards > > Jeff Zhang >
Re: Building Spark 2.X in Intellij
It works well with me. You can try reimport it into intellij. On Thu, Jun 23, 2016 at 10:25 AM, Stephen Boesch wrote: > > Building inside intellij is an ever moving target. Anyone have the magical > procedures to get it going for 2.X? > > There are numerous library references that - although included in the > pom.xml build - are for some reason not found when processed within > Intellij. > -- Best Regards Jeff Zhang
Building Spark 2.X in Intellij
Building inside intellij is an ever moving target. Anyone have the magical procedures to get it going for 2.X? There are numerous library references that - although included in the pom.xml build - are for some reason not found when processed within Intellij.