I have to try but I think it probably will happen with spark-shell as well.
I have found alternative to pass a Array to UDF as a parameter.
Thanks
On Thu, Nov 17, 2016 at 9:24 AM, moon soo Lee <m...@apache.org> wrote:
> Are you able to run the same code in SPARK_HOME/bin/spark-shell ?
>
> Thanks,
> moon
>
> On Mon, Nov 14, 2016 at 2:21 PM Nirav Patel <npa...@xactlycorp.com> wrote:
>
>> I have a following map:
>>
>> final val idxMapArr = idxMap.collectAsMap
>>
>> Which is being used in one of spark transformation here:
>>
>> def labelStr(predictions: WrappedArray[Double]): Array[String] = {
>>
>> predictions.array.map(*idxMapArr*.getOrElse(_, "Other"))
>> }
>>
>> I get following error :
>>
>>
>> Caused by: java.io.NotSerializableException: com.github.fommil.netlib.
>> F2jBLAS
>> Serialization stack:
>> - object not serializable (class: com.github.fommil.netlib.F2jBLAS,
>> value: com.github.fommil.netlib.F2jBLAS@294770d3)
>> - field (class: org.apache.spark.ml.tuning.CrossValidator, name:
>> f2jBLAS, type: class com.github.fommil.netlib.F2jBLAS)
>> - object (class org.apache.spark.ml.tuning.CrossValidator,
>> cv_891fd6b7d95f)
>> - field (class: $iw, name: crossValidator, type: class
>> org.apache.spark.ml.tuning.CrossValidator)
>> - object (class $iw, $iw@556a6aed)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4a1212a0)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@34f1718d)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@1d4bb5f7)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@5a30765)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@39a33f4)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@335d6de3)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@2b4ccb90)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@1814c6c9)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@7cd91fb0)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@d6c803f)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@488b6cf6)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4eee8af2)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4301fc7)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@5dd61af2)
>> - field (class: $iw, name: $iw, type: class $iw)
>> - object (class $iw, $iw@4ce26a93)
>> - field (class: $line95653076098.$read, name: $iw, type: class $iw)
>> - object (class $line95653076098.$read, $line95653076098.$read@c7d550f)
>> - field (class: $iw, name: $line95653076098$read, type: class
>> $line95653076098.$read)
>> - object (class $iw, $iw@1c6701cd)
>>
>>
>>
>> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
>>
>> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
>> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
>> <https://twitter.com/Xactly> [image: Facebook]
>> <https://www.facebook.com/XactlyCorp> [image: YouTube]
>> <http://www.youtube.com/xactlycorporation>
>
>
--
[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>
<https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn]
<https://www.linkedin.com/company/xactly-corporation> [image: Twitter]
<https://twitter.com/Xactly> [image: Facebook]
<https://www.facebook.com/XactlyCorp> [image: YouTube]
<http://www.youtube.com/xactlycorporation>
