~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-slave.sh spark://
192.168.0.38:7077
~/spark-3.0.0-preview2-bin-hadoop2.7$ sbin/start-master.sh
Backbutton.co.uk
¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org
<http://www.backbutton.co.uk>
On Fri, 27 Mar 2020 at 06:12, Zahid Rahman <[email protected]> wrote:
> sbin/start-master.sh
> sbin/start-slave.sh spark://192.168.0.38:7077
>
> Backbutton.co.uk
> ¯\_(ツ)_/¯
> ♡۶Java♡۶RMI ♡۶
> Make Use Method {MUM}
> makeuse.org
> <http://www.backbutton.co.uk>
>
>
> On Fri, 27 Mar 2020 at 05:59, Wenchen Fan <[email protected]> wrote:
>
>> Your Spark cluster, spark://192.168.0.38:7077, how is it deployed if you
>> just include Spark dependency in IntelliJ?
>>
>> On Fri, Mar 27, 2020 at 1:54 PM Zahid Rahman <[email protected]>
>> wrote:
>>
>>> I have configured in IntelliJ as external jars
>>> spark-3.0.0-preview2-bin-hadoop2.7/jar
>>>
>>> not pulling anything from maven.
>>>
>>> Backbutton.co.uk
>>> ¯\_(ツ)_/¯
>>> ♡۶Java♡۶RMI ♡۶
>>> Make Use Method {MUM}
>>> makeuse.org
>>> <http://www.backbutton.co.uk>
>>>
>>>
>>> On Fri, 27 Mar 2020 at 05:45, Wenchen Fan <[email protected]> wrote:
>>>
>>>> Which Spark/Scala version do you use?
>>>>
>>>> On Fri, Mar 27, 2020 at 1:24 PM Zahid Rahman <[email protected]>
>>>> wrote:
>>>>
>>>>>
>>>>> with the following sparksession configuration
>>>>>
>>>>> val spark = SparkSession.builder().master("local[*]").appName("Spark
>>>>> Session take").getOrCreate();
>>>>>
>>>>> this line works
>>>>>
>>>>> flights.filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME !=
>>>>> "Canada").map(flight_row => flight_row).take(5)
>>>>>
>>>>>
>>>>> however if change the master url like so, with the ip address then the
>>>>> following error is produced by the position of .take(5)
>>>>>
>>>>> val spark =
>>>>> SparkSession.builder().master("spark://192.168.0.38:7077").appName("Spark
>>>>> Session take").getOrCreate();
>>>>>
>>>>>
>>>>> 20/03/27 05:15:20 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID
>>>>> 1, 192.168.0.38, executor 0): java.lang.ClassCastException: cannot assign
>>>>> instance of java.lang.invoke.SerializedLambda to field
>>>>> org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in
>>>>> instance
>>>>> of org.apache.spark.rdd.MapPartitionsRDD
>>>>>
>>>>> BUT if I remove take(5) or change the position of take(5) or insert
>>>>> an extra take(5) as illustrated in code then it works. I don't see why the
>>>>> position of take(5) should cause such an error or be caused by changing
>>>>> the
>>>>> master url
>>>>>
>>>>> flights.take(5).filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME !=
>>>>> "Canada").map(flight_row => flight_row).take(5)
>>>>>
>>>>> flights.take(5)
>>>>>
>>>>> flights
>>>>> .take(5)
>>>>> .filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME != "Canada")
>>>>> .map(fr => flight(fr.DEST_COUNTRY_NAME, fr.ORIGIN_COUNTRY_NAME,fr.count
>>>>> + 5))
>>>>> flights.show(5)
>>>>>
>>>>>
>>>>> complete code if you wish to replicate it.
>>>>>
>>>>> import org.apache.spark.sql.SparkSession
>>>>>
>>>>> object sessiontest {
>>>>>
>>>>> // define specific data type class then manipulate it using the filter
>>>>> and map functions
>>>>> // this is also known as an Encoder
>>>>> case class flight (DEST_COUNTRY_NAME: String,
>>>>> ORIGIN_COUNTRY_NAME:String,
>>>>> count: BigInt)
>>>>>
>>>>>
>>>>> def main(args:Array[String]): Unit ={
>>>>>
>>>>> val spark =
>>>>> SparkSession.builder().master("spark://192.168.0.38:7077").appName("Spark
>>>>> Session take").getOrCreate();
>>>>>
>>>>> import spark.implicits._
>>>>> val flightDf =
>>>>> spark.read.parquet("/data/flight-data/parquet/2010-summary.parquet/")
>>>>> val flights = flightDf.as[flight]
>>>>>
>>>>> flights.take(5).filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME
>>>>> != "Canada").map(flight_row => flight_row).take(5)
>>>>>
>>>>> flights.take(5)
>>>>>
>>>>> flights
>>>>> .take(5)
>>>>> .filter(flight_row => flight_row.ORIGIN_COUNTRY_NAME != "Canada")
>>>>> .map(fr => flight(fr.DEST_COUNTRY_NAME,
>>>>> fr.ORIGIN_COUNTRY_NAME,fr.count + 5))
>>>>> flights.show(5)
>>>>>
>>>>> } // main
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Backbutton.co.uk
>>>>> ¯\_(ツ)_/¯
>>>>> ♡۶Java♡۶RMI ♡۶
>>>>> Make Use Method {MUM}
>>>>> makeuse.org
>>>>> <http://www.backbutton.co.uk>
>>>>>
>>>>
