RE: NPE when using Joda DateTime
OK, this isn’t very efficient, but I’ve found a solution: I am still passing around joda DateTime’s – however if I ever want to use any of the minusHours/minusDays etc “transient” variables, I simply do this: var tempDate = new org.joda.time.DateTime(transaction.date.toInstant) then: tempDate.minusHours(1) works fine. Kind Regards, Alex. From: Romain Sagean [mailto:romain.sag...@hupi.fr] Sent: 15 January 2016 13:08 To: Sean Owen Cc: Spencer, Alex (Santander); Shixiong(Ryan) Zhu; user@spark.apache.org Subject: Re: NPE when using Joda DateTime Hi, I had a similar problem with Joda Time though i didn't use Kryo, the solution I found was to use standard java date and time classes instead of Joda. 2016-01-15 13:16 GMT+01:00 Sean Owen mailto:so...@cloudera.com>>: I haven't dug into this, but I agree that something that is transient isn't meant to be restored by the default Java serialization mechanism. I'd expect the class handles restoring that value as needed or in a custom readObject method. And then I don't know how Kryo interacts with that. I don't think you need to install anything. You may end up writing your own serialization for Kryo. Try not using Kryo just to narrow it down? Hackier solution: send around long timestamps and then make them into DateTime locally as needed. Not great. Or if possible use Java 8, where Joda APIs are part of the JDK. Possibly it works then. On Fri, Jan 15, 2016 at 9:32 AM, Spencer, Alex (Santander) mailto:alex.spen...@santander.co.uk>> wrote: > Hi, > > > > I tried Zhu’s recommendation and sadly got the same error. (Again, single > map worked by the groupBy / flatMap generates this error). > > > > Does Kryo has a bug i.e. it’s not serialising all components needed, or do > I just need to get our IT team to install those magro Serializers as > suggested by Todd? If that variable is transient then actually that means > Kryo is working as it’s meant to? > > > > Am I at the point where I should pull apart the source code and build my own > DateTime class? I hate reinventing the wheel though. > > > > Thanks, > > Alex. > > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org> -- Romain Sagean romain.sag...@hupi.fr<mailto:romain.sag...@hupi.fr> [cid:image001.png@01D14F9A.B07105D0] Emails aren't always secure, and they may be intercepted or changed after they've been sent. Santander doesn't accept liability if this happens. If you think someone may have interfered with this email, please get in touch with the sender another way. This message doesn't create or change any contract. Santander doesn't accept responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. If you've received this email by mistake, please let the sender know at once that it's gone to the wrong person and then destroy it without copying, using, or telling anyone about its contents. Santander UK plc Reg. No. 2294747 and Abbey National Treasury Services plc Reg. No. 2338548 Registered Offices: 2 Triton Square, Regent's Place, London NW1 3AN. Registered in England. www.santander.co.uk. Authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. FCA Reg. No. 106054 and 146003 respectively. Santander Sharedealing is a trading name of Abbey Stockbrokers Limited Reg. No. 02666793. Registered Office: Kingfisher House, Radford Way, Billericay, Essex CM12 0GZ. Authorised and regulated by the Financial Conduct Authority. FCA Reg. No. 154210. You can check this on the Financial Services Register by visiting the FCA’s website www.fca.org.uk/register or by contacting the FCA on 0800 111 6768. Santander UK plc is also licensed by the Financial Supervision Commission of the Isle of Man for its branch in the Isle of Man. Deposits held with the Isle of Man branch are covered by the Isle of Man Depositors’ Compensation Scheme as set out in the Isle of Man Depositors’ Compensation Scheme Regulations 2010. In the Isle of Man, Santander UK plc’s principal place of business is at 19/21 Prospect Hill, Douglas, Isle of Man, IM1 1ET. Santander and the flame logo are registered trademarks. Santander Asset Finance plc. Reg. No. 1533123. Registered Office: 2 Triton Square, Regent’s Place, London NW1 3AN. Registered in England. Santander Corporate & Commercial is a brand name used by Santander UK plc, Abbey National Treasury Services plc and Santander Asset Finance plc. Ref:[PDB#1-4A]
Re: NPE when using Joda DateTime
Hi, I had a similar problem with Joda Time though i didn't use Kryo, the solution I found was to use standard java date and time classes instead of Joda. 2016-01-15 13:16 GMT+01:00 Sean Owen : > I haven't dug into this, but I agree that something that is transient > isn't meant to be restored by the default Java serialization > mechanism. I'd expect the class handles restoring that value as needed > or in a custom readObject method. And then I don't know how Kryo > interacts with that. > > I don't think you need to install anything. You may end up writing > your own serialization for Kryo. > > Try not using Kryo just to narrow it down? > > Hackier solution: send around long timestamps and then make them into > DateTime locally as needed. Not great. > > Or if possible use Java 8, where Joda APIs are part of the JDK. > Possibly it works then. > > On Fri, Jan 15, 2016 at 9:32 AM, Spencer, Alex (Santander) > wrote: > > Hi, > > > > > > > > I tried Zhu’s recommendation and sadly got the same error. (Again, single > > map worked by the groupBy / flatMap generates this error). > > > > > > > > Does Kryo has a bug i.e. it’s not serialising all components needed, or > do > > I just need to get our IT team to install those magro Serializers as > > suggested by Todd? If that variable is transient then actually that means > > Kryo is working as it’s meant to? > > > > > > > > Am I at the point where I should pull apart the source code and build my > own > > DateTime class? I hate reinventing the wheel though. > > > > > > > > Thanks, > > > > Alex. > > > > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- *Romain Sagean* *romain.sag...@hupi.fr *
RE: NPE when using Joda DateTime
I'll try the hackier way for now - given the limitation of not being able to modify the environment we've been given. Thanks all for your help so far. Kind Regards, Alex. -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: 15 January 2016 12:17 To: Spencer, Alex (Santander) Cc: Shixiong(Ryan) Zhu; user@spark.apache.org Subject: Re: NPE when using Joda DateTime I haven't dug into this, but I agree that something that is transient isn't meant to be restored by the default Java serialization mechanism. I'd expect the class handles restoring that value as needed or in a custom readObject method. And then I don't know how Kryo interacts with that. I don't think you need to install anything. You may end up writing your own serialization for Kryo. Try not using Kryo just to narrow it down? Hackier solution: send around long timestamps and then make them into DateTime locally as needed. Not great. Or if possible use Java 8, where Joda APIs are part of the JDK. Possibly it works then. On Fri, Jan 15, 2016 at 9:32 AM, Spencer, Alex (Santander) wrote: > Hi, > > > > I tried Zhu’s recommendation and sadly got the same error. (Again, > single map worked by the groupBy / flatMap generates this error). > > > > Does Kryo has a bug i.e. it’s not serialising all components needed, > or do I just need to get our IT team to install those magro > Serializers as suggested by Todd? If that variable is transient then > actually that means Kryo is working as it’s meant to? > > > > Am I at the point where I should pull apart the source code and build > my own DateTime class? I hate reinventing the wheel though. > > > > Thanks, > > Alex. > > Emails aren't always secure, and they may be intercepted or changed after they've been sent. Santander doesn't accept liability if this happens. If you think someone may have interfered with this email, please get in touch with the sender another way. This message doesn't create or change any contract. Santander doesn't accept responsibility for damage caused by any viruses contained in this email or its attachments. Emails may be monitored. If you've received this email by mistake, please let the sender know at once that it's gone to the wrong person and then destroy it without copying, using, or telling anyone about its contents. Santander UK plc Reg. No. 2294747 and Abbey National Treasury Services plc Reg. No. 2338548 Registered Offices: 2 Triton Square, Regent's Place, London NW1 3AN. Registered in England. www.santander.co.uk. Authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. FCA Reg. No. 106054 and 146003 respectively. Santander Sharedealing is a trading name of Abbey Stockbrokers Limited Reg. No. 02666793. Registered Office: Kingfisher House, Radford Way, Billericay, Essex CM12 0GZ. Authorised and regulated by the Financial Conduct Authority. FCA Reg. No. 154210. You can check this on the Financial Services Register by visiting the FCA’s website www.fca.org.uk/register or by contacting the FCA on 0800 111 6768. Santander UK plc is also licensed by the Financial Supervision Commission of the Isle of Man for its branch in the Isle of Man. Deposits held with the Isle of Man branch are covered by the Isle of Man Depositors’ Compensation Scheme as set out in the Isle of Man Depositors’ Compensation Scheme Regulations 2010. In the Isle of Man, Santander UK plc’s principal place of business is at 19/21 Prospect Hill, Douglas, Isle of Man, IM1 1ET. Santander and the flame logo are registered trademarks. Santander Asset Finance plc. Reg. No. 1533123. Registered Office: 2 Triton Square, Regent’s Place, London NW1 3AN. Registered in England. Santander Corporate & Commercial is a brand name used by Santander UK plc, Abbey National Treasury Services plc and Santander Asset Finance plc. Ref:[PDB#1-4A]
Re: NPE when using Joda DateTime
I haven't dug into this, but I agree that something that is transient isn't meant to be restored by the default Java serialization mechanism. I'd expect the class handles restoring that value as needed or in a custom readObject method. And then I don't know how Kryo interacts with that. I don't think you need to install anything. You may end up writing your own serialization for Kryo. Try not using Kryo just to narrow it down? Hackier solution: send around long timestamps and then make them into DateTime locally as needed. Not great. Or if possible use Java 8, where Joda APIs are part of the JDK. Possibly it works then. On Fri, Jan 15, 2016 at 9:32 AM, Spencer, Alex (Santander) wrote: > Hi, > > > > I tried Zhu’s recommendation and sadly got the same error. (Again, single > map worked by the groupBy / flatMap generates this error). > > > > Does Kryo has a bug i.e. it’s not serialising all components needed, or do > I just need to get our IT team to install those magro Serializers as > suggested by Todd? If that variable is transient then actually that means > Kryo is working as it’s meant to? > > > > Am I at the point where I should pull apart the source code and build my own > DateTime class? I hate reinventing the wheel though. > > > > Thanks, > > Alex. > > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
RE: NPE when using Joda DateTime
Hi,
I tried Zhu’s recommendation and sadly got the same error. (Again, single map
worked by the groupBy / flatMap generates this error).
Does Kryo has a bug i.e. it’s not serialising all components needed, or do I
just need to get our IT team to install those magro Serializers as suggested by
Todd? If that variable is transient then actually that means Kryo is working as
it’s meant to?
Am I at the point where I should pull apart the source code and build my own
DateTime class? I hate reinventing the wheel though.
Thanks,
Alex.
From: Shixiong(Ryan) Zhu [mailto:shixi...@databricks.com]
Sent: 14 January 2016 21:57
To: Durgesh Verma
Cc: Spencer, Alex (Santander); Todd Nist; Sean Owen; user@spark.apache.org
Subject: Re: NPE when using Joda DateTime
Could you try to use "Kryo.setDefaultSerializer" like this:
class YourKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.setDefaultSerializer(classOf[com.esotericsoftware.kryo.serializers.JavaSerializer])
}
}
On Thu, Jan 14, 2016 at 12:54 PM, Durgesh Verma
mailto:dv21...@gmail.com>> wrote:
Today is my day... Trying to go thru where I can pitch in. Let me know if below
makes sense.
I looked at joda Java Api source code (1.2.9) and traced that call in NPE. It
looks like AssembledChronology class is being used, the iYears instance
variable is defined as transient.
DateTime.minusYears(int years) call trace:
long instant = getChronology().years().subtract(getMillis(), years);
Not sure how the suggested serializer would help if variable is transient.
Thanks,
-Durgesh
On Jan 14, 2016, at 11:49 AM, Spencer, Alex (Santander)
mailto:alex.spen...@santander.co.uk.invalid>>
wrote:
I appreciate this – thank you.
I’m not an admin on the box I’m using spark-shell on – so I’m not sure I can
add them to that namespace. I’m hoping if I declare the JodaDateTimeSerializer
class in my REPL that I can still get this to work. I think the INTERVAL part
below may be key, I haven’t tried that yet.
Kind Regards,
Alex.
From: Todd Nist [mailto:tsind...@gmail.com]
Sent: 14 January 2016 16:28
To: Spencer, Alex (Santander)
Cc: Sean Owen; user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: NPE when using Joda DateTime
I had a similar problem a while back and leveraged these Kryo serializers,
https://github.com/magro/kryo-serializers. I had to fallback to version 0.28,
but that was a while back. You can add these to the
org.apache.spark.serializer.KryoRegistrator
and then set your registrator in the spark config:
sparkConfig.
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", "com.yourpackage.YourKryoRegistrator")
...
where YourKryoRegistrator is something like:
class YourKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[org.joda.time.DateTime], new JodaDateTimeSerializer)
kryo.register(classOf[org.joda.time.Interval], new JodaIntervalSerializer)
}
}
HTH.
-Todd
On Thu, Jan 14, 2016 at 9:28 AM, Spencer, Alex (Santander)
mailto:alex.spen...@santander.co.uk.invalid>>
wrote:
Hi,
I tried take(1500) and test.collect and these both work on the "single" map
statement.
I'm very new to Kryo serialisation, I managed to find some code and I copied
and pasted and that's what originally made the single map statement work:
class MyRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[org.joda.time.DateTime])
}
}
Is it because the groupBy sees a different class type? Maybe Array[DateTime]? I
don’t want to find the answer by trial and error though.
Alex
-Original Message-
From: Sean Owen [mailto:so...@cloudera.com<mailto:so...@cloudera.com>]
Sent: 14 January 2016 14:07
To: Spencer, Alex (Santander)
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: NPE when using Joda DateTime
It does look somehow like the state of the DateTime object isn't being
recreated properly on deserialization somehow, given where the NPE occurs (look
at the Joda source code). However the object is java.io.Serializable. Are you
sure the Kryo serialization is correct?
It doesn't quite explain why the map operation works by itself. It could be the
difference between executing locally (take(1) will look at 1 partition in 1
task which prefers to be local) and executing remotely (groupBy is going to
need a shuffle).
On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander)
mailto:alex.spen...@santander.co.uk.invalid>>
wrote:
> Hello,
>
>
>
> I was wondering if somebody is able to help me get to the bottom of a
> null pointer exception I’m seeing in my code. I’ve managed to narrow
> down a problem in a larger class to my use of Joda’s DateTime
> functions. I’ve success
Re: NPE when using Joda DateTime
Could you try to use "Kryo.setDefaultSerializer" like this:
class YourKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.setDefaultSerializer(classOf[com.esotericsoftware.kryo.serializers.JavaSerializer])
}
}
On Thu, Jan 14, 2016 at 12:54 PM, Durgesh Verma wrote:
> Today is my day... Trying to go thru where I can pitch in. Let me know if
> below makes sense.
>
> I looked at joda Java Api source code (1.2.9) and traced that call in NPE.
> It looks like AssembledChronology class is being used, the iYears instance
> variable is defined as transient.
>
> DateTime.minusYears(int years) call trace:
> long instant = getChronology().years().subtract(getMillis(), years);
>
> Not sure how the suggested serializer would help if variable is transient.
>
> Thanks,
> -Durgesh
>
> On Jan 14, 2016, at 11:49 AM, Spencer, Alex (Santander) <
> alex.spen...@santander.co.uk.INVALID
> > wrote:
>
> I appreciate this – thank you.
>
>
>
> I’m not an admin on the box I’m using spark-shell on – so I’m not sure I
> can add them to that namespace. I’m hoping if I declare the
> JodaDateTimeSerializer class in my REPL that I can still get this to work.
> I think the INTERVAL part below may be key, I haven’t tried that yet.
>
>
>
> Kind Regards,
>
> Alex.
>
>
>
> *From:* Todd Nist [mailto:tsind...@gmail.com ]
> *Sent:* 14 January 2016 16:28
> *To:* Spencer, Alex (Santander)
> *Cc:* Sean Owen; user@spark.apache.org
> *Subject:* Re: NPE when using Joda DateTime
>
>
>
> I had a similar problem a while back and leveraged these Kryo serializers,
> https://github.com/magro/kryo-serializers. I had to fallback to version
> 0.28, but that was a while back. You can add these to the
>
> org.apache.spark.serializer.KryoRegistrator
>
> and then set your registrator in the spark config:
>
> sparkConfig.
> .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> .set("spark.kryo.registrator", "com.yourpackage.YourKryoRegistrator")
> ...
>
> where YourKryoRegistrator is something like:
>
> class YourKryoRegistrator extends KryoRegistrator {
>
> override def registerClasses(kryo: Kryo) {
> kryo.register(classOf[org.joda.time.DateTime], new
> JodaDateTimeSerializer)
> kryo.register(classOf[org.joda.time.Interval], new
> JodaIntervalSerializer)
> }
> }
>
> HTH.
>
> -Todd
>
>
>
> On Thu, Jan 14, 2016 at 9:28 AM, Spencer, Alex (Santander) <
> alex.spen...@santander.co.uk.invalid> wrote:
>
> Hi,
>
> I tried take(1500) and test.collect and these both work on the "single"
> map statement.
>
> I'm very new to Kryo serialisation, I managed to find some code and I
> copied and pasted and that's what originally made the single map statement
> work:
>
> class MyRegistrator extends KryoRegistrator {
> override def registerClasses(kryo: Kryo) {
> kryo.register(classOf[org.joda.time.DateTime])
> }
> }
>
> Is it because the groupBy sees a different class type? Maybe
> Array[DateTime]? I don’t want to find the answer by trial and error though.
>
> Alex
>
> -Original Message-
> From: Sean Owen [mailto:so...@cloudera.com]
> Sent: 14 January 2016 14:07
> To: Spencer, Alex (Santander)
> Cc: user@spark.apache.org
> Subject: Re: NPE when using Joda DateTime
>
> It does look somehow like the state of the DateTime object isn't being
> recreated properly on deserialization somehow, given where the NPE occurs
> (look at the Joda source code). However the object is java.io.Serializable.
> Are you sure the Kryo serialization is correct?
>
> It doesn't quite explain why the map operation works by itself. It could
> be the difference between executing locally (take(1) will look at 1
> partition in 1 task which prefers to be local) and executing remotely
> (groupBy is going to need a shuffle).
>
> On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander) <
> alex.spen...@santander.co.uk.invalid> wrote:
> > Hello,
> >
> >
> >
> > I was wondering if somebody is able to help me get to the bottom of a
> > null pointer exception I’m seeing in my code. I’ve managed to narrow
> > down a problem in a larger class to my use of Joda’s DateTime
> > functions. I’ve successfully run my code in scala, but I’ve hit a few
> > problems when adapting it to run in spark.
> >
> >
> >
> > Spark version: 1.3.0
> >
> > Scala version: 2.10.4
> >
> > Java HotSpot 1.7
> >
> >
> >
> > I have a small case class called Transacti
Re: NPE when using Joda DateTime
Today is my day... Trying to go thru where I can pitch in. Let me know if below
makes sense.
I looked at joda Java Api source code (1.2.9) and traced that call in NPE. It
looks like AssembledChronology class is being used, the iYears instance
variable is defined as transient.
DateTime.minusYears(int years) call trace:
long instant = getChronology().years().subtract(getMillis(), years);
Not sure how the suggested serializer would help if variable is transient.
Thanks,
-Durgesh
> On Jan 14, 2016, at 11:49 AM, Spencer, Alex (Santander)
> wrote:
>
> I appreciate this – thank you.
>
> I’m not an admin on the box I’m using spark-shell on – so I’m not sure I can
> add them to that namespace. I’m hoping if I declare the
> JodaDateTimeSerializer class in my REPL that I can still get this to work. I
> think the INTERVAL part below may be key, I haven’t tried that yet.
>
> Kind Regards,
> Alex.
>
> From: Todd Nist [mailto:tsind...@gmail.com]
> Sent: 14 January 2016 16:28
> To: Spencer, Alex (Santander)
> Cc: Sean Owen; user@spark.apache.org
> Subject: Re: NPE when using Joda DateTime
>
> I had a similar problem a while back and leveraged these Kryo serializers,
> https://github.com/magro/kryo-serializers. I had to fallback to version
> 0.28, but that was a while back. You can add these to the
> org.apache.spark.serializer.KryoRegistrator
> and then set your registrator in the spark config:
>
> sparkConfig.
> .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> .set("spark.kryo.registrator", "com.yourpackage.YourKryoRegistrator")
> ...
>
> where YourKryoRegistrator is something like:
>
> class YourKryoRegistrator extends KryoRegistrator {
> override def registerClasses(kryo: Kryo) {
> kryo.register(classOf[org.joda.time.DateTime], new JodaDateTimeSerializer)
> kryo.register(classOf[org.joda.time.Interval], new JodaIntervalSerializer)
> }
> }
> HTH.
> -Todd
>
> On Thu, Jan 14, 2016 at 9:28 AM, Spencer, Alex (Santander)
> wrote:
> Hi,
>
> I tried take(1500) and test.collect and these both work on the "single" map
> statement.
>
> I'm very new to Kryo serialisation, I managed to find some code and I copied
> and pasted and that's what originally made the single map statement work:
>
> class MyRegistrator extends KryoRegistrator {
> override def registerClasses(kryo: Kryo) {
> kryo.register(classOf[org.joda.time.DateTime])
> }
> }
>
> Is it because the groupBy sees a different class type? Maybe Array[DateTime]?
> I don’t want to find the answer by trial and error though.
>
> Alex
>
> -Original Message-
> From: Sean Owen [mailto:so...@cloudera.com]
> Sent: 14 January 2016 14:07
> To: Spencer, Alex (Santander)
> Cc: user@spark.apache.org
> Subject: Re: NPE when using Joda DateTime
>
> It does look somehow like the state of the DateTime object isn't being
> recreated properly on deserialization somehow, given where the NPE occurs
> (look at the Joda source code). However the object is java.io.Serializable.
> Are you sure the Kryo serialization is correct?
>
> It doesn't quite explain why the map operation works by itself. It could be
> the difference between executing locally (take(1) will look at 1 partition in
> 1 task which prefers to be local) and executing remotely (groupBy is going to
> need a shuffle).
>
> On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander)
> wrote:
> > Hello,
> >
> >
> >
> > I was wondering if somebody is able to help me get to the bottom of a
> > null pointer exception I’m seeing in my code. I’ve managed to narrow
> > down a problem in a larger class to my use of Joda’s DateTime
> > functions. I’ve successfully run my code in scala, but I’ve hit a few
> > problems when adapting it to run in spark.
> >
> >
> >
> > Spark version: 1.3.0
> >
> > Scala version: 2.10.4
> >
> > Java HotSpot 1.7
> >
> >
> >
> > I have a small case class called Transaction, which looks something
> > like
> > this:
> >
> >
> >
> > case class Transaction(date : org.joda.time.DateTime = new
> > org.joda.time.DateTime())
> >
> >
> >
> > I have an RDD[Transactions] trans:
> >
> > org.apache.spark.rdd.RDD[Transaction] = MapPartitionsRDD[4] at map at
> > :44
> >
> >
> >
> > I am able to run this successfully:
> >
> >
> >
> > val test = trans.map(_.date.minusYears(10))
> >
> > test.take(1)
> >
> >
> >
> > Ho
RE: NPE when using Joda DateTime
I appreciate this – thank you.
I’m not an admin on the box I’m using spark-shell on – so I’m not sure I can
add them to that namespace. I’m hoping if I declare the JodaDateTimeSerializer
class in my REPL that I can still get this to work. I think the INTERVAL part
below may be key, I haven’t tried that yet.
Kind Regards,
Alex.
From: Todd Nist [mailto:tsind...@gmail.com]
Sent: 14 January 2016 16:28
To: Spencer, Alex (Santander)
Cc: Sean Owen; user@spark.apache.org
Subject: Re: NPE when using Joda DateTime
I had a similar problem a while back and leveraged these Kryo serializers,
https://github.com/magro/kryo-serializers. I had to fallback to version 0.28,
but that was a while back. You can add these to the
org.apache.spark.serializer.KryoRegistrator
and then set your registrator in the spark config:
sparkConfig.
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", "com.yourpackage.YourKryoRegistrator")
...
where YourKryoRegistrator is something like:
class YourKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[org.joda.time.DateTime], new JodaDateTimeSerializer)
kryo.register(classOf[org.joda.time.Interval], new JodaIntervalSerializer)
}
}
HTH.
-Todd
On Thu, Jan 14, 2016 at 9:28 AM, Spencer, Alex (Santander)
mailto:alex.spen...@santander.co.uk.invalid>>
wrote:
Hi,
I tried take(1500) and test.collect and these both work on the "single" map
statement.
I'm very new to Kryo serialisation, I managed to find some code and I copied
and pasted and that's what originally made the single map statement work:
class MyRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[org.joda.time.DateTime])
}
}
Is it because the groupBy sees a different class type? Maybe Array[DateTime]? I
don’t want to find the answer by trial and error though.
Alex
-Original Message-
From: Sean Owen [mailto:so...@cloudera.com<mailto:so...@cloudera.com>]
Sent: 14 January 2016 14:07
To: Spencer, Alex (Santander)
Cc: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Re: NPE when using Joda DateTime
It does look somehow like the state of the DateTime object isn't being
recreated properly on deserialization somehow, given where the NPE occurs (look
at the Joda source code). However the object is java.io.Serializable. Are you
sure the Kryo serialization is correct?
It doesn't quite explain why the map operation works by itself. It could be the
difference between executing locally (take(1) will look at 1 partition in 1
task which prefers to be local) and executing remotely (groupBy is going to
need a shuffle).
On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander)
mailto:alex.spen...@santander.co.uk.invalid>>
wrote:
> Hello,
>
>
>
> I was wondering if somebody is able to help me get to the bottom of a
> null pointer exception I’m seeing in my code. I’ve managed to narrow
> down a problem in a larger class to my use of Joda’s DateTime
> functions. I’ve successfully run my code in scala, but I’ve hit a few
> problems when adapting it to run in spark.
>
>
>
> Spark version: 1.3.0
>
> Scala version: 2.10.4
>
> Java HotSpot 1.7
>
>
>
> I have a small case class called Transaction, which looks something
> like
> this:
>
>
>
> case class Transaction(date : org.joda.time.DateTime = new
> org.joda.time.DateTime())
>
>
>
> I have an RDD[Transactions] trans:
>
> org.apache.spark.rdd.RDD[Transaction] = MapPartitionsRDD[4] at map at
> :44
>
>
>
> I am able to run this successfully:
>
>
>
> val test = trans.map(_.date.minusYears(10))
>
> test.take(1)
>
>
>
> However if I do:
>
>
>
> val groupedTrans = trans.groupBy(_.account)
>
>
>
> //For each group, process transactions in turn:
>
> val test = groupedTrans.flatMap { case (_, transList) =>
>
> transList.map {transaction =>
>
> transaction.date.minusYears(10)
>
> }
>
> }
>
> test.take(1)
>
>
>
> I get:
>
>
>
> java.lang.NullPointerException
>
> at org.joda.time.DateTime.minusYears(DateTime.java:1268)
>
>
>
> Should the second operation not be equivalent to the first .map one?
> (It’s a long way round of producing my error – but it’s extremely
> similar to what’s happening in my class).
>
>
>
> I’ve got a custom registration class for Kryo which I think is working
> - before I added this the original .map did not work – but shouldn’t
> it be able to serialize all instances of Joda DateTime?
>
>
>
> Thank you for any help / pointers you can give me.
>
>
>
> Kind Regard
Re: NPE when using Joda DateTime
I had a similar problem a while back and leveraged these Kryo serializers,
https://github.com/magro/kryo-serializers. I had to fallback to version
0.28, but that was a while back. You can add these to the
org.apache.spark.serializer.KryoRegistrator
and then set your registrator in the spark config:
sparkConfig.
.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator", "com.yourpackage.YourKryoRegistrator")
...
where YourKryoRegistrator is something like:
class YourKryoRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[org.joda.time.DateTime], new
JodaDateTimeSerializer)
kryo.register(classOf[org.joda.time.Interval], new
JodaIntervalSerializer)
}
}
HTH.
-Todd
On Thu, Jan 14, 2016 at 9:28 AM, Spencer, Alex (Santander) <
alex.spen...@santander.co.uk.invalid> wrote:
> Hi,
>
> I tried take(1500) and test.collect and these both work on the "single"
> map statement.
>
> I'm very new to Kryo serialisation, I managed to find some code and I
> copied and pasted and that's what originally made the single map statement
> work:
>
> class MyRegistrator extends KryoRegistrator {
> override def registerClasses(kryo: Kryo) {
> kryo.register(classOf[org.joda.time.DateTime])
> }
> }
>
> Is it because the groupBy sees a different class type? Maybe
> Array[DateTime]? I don’t want to find the answer by trial and error though.
>
> Alex
>
> -Original Message-
> From: Sean Owen [mailto:so...@cloudera.com]
> Sent: 14 January 2016 14:07
> To: Spencer, Alex (Santander)
> Cc: user@spark.apache.org
> Subject: Re: NPE when using Joda DateTime
>
> It does look somehow like the state of the DateTime object isn't being
> recreated properly on deserialization somehow, given where the NPE occurs
> (look at the Joda source code). However the object is java.io.Serializable.
> Are you sure the Kryo serialization is correct?
>
> It doesn't quite explain why the map operation works by itself. It could
> be the difference between executing locally (take(1) will look at 1
> partition in 1 task which prefers to be local) and executing remotely
> (groupBy is going to need a shuffle).
>
> On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander)
> wrote:
> > Hello,
> >
> >
> >
> > I was wondering if somebody is able to help me get to the bottom of a
> > null pointer exception I’m seeing in my code. I’ve managed to narrow
> > down a problem in a larger class to my use of Joda’s DateTime
> > functions. I’ve successfully run my code in scala, but I’ve hit a few
> > problems when adapting it to run in spark.
> >
> >
> >
> > Spark version: 1.3.0
> >
> > Scala version: 2.10.4
> >
> > Java HotSpot 1.7
> >
> >
> >
> > I have a small case class called Transaction, which looks something
> > like
> > this:
> >
> >
> >
> > case class Transaction(date : org.joda.time.DateTime = new
> > org.joda.time.DateTime())
> >
> >
> >
> > I have an RDD[Transactions] trans:
> >
> > org.apache.spark.rdd.RDD[Transaction] = MapPartitionsRDD[4] at map at
> > :44
> >
> >
> >
> > I am able to run this successfully:
> >
> >
> >
> > val test = trans.map(_.date.minusYears(10))
> >
> > test.take(1)
> >
> >
> >
> > However if I do:
> >
> >
> >
> > val groupedTrans = trans.groupBy(_.account)
> >
> >
> >
> > //For each group, process transactions in turn:
> >
> > val test = groupedTrans.flatMap { case (_, transList) =>
> >
> > transList.map {transaction =>
> >
> > transaction.date.minusYears(10)
> >
> > }
> >
> > }
> >
> > test.take(1)
> >
> >
> >
> > I get:
> >
> >
> >
> > java.lang.NullPointerException
> >
> > at org.joda.time.DateTime.minusYears(DateTime.java:1268)
> >
> >
> >
> > Should the second operation not be equivalent to the first .map one?
> > (It’s a long way round of producing my error – but it’s extremely
> > similar to what’s happening in my class).
> >
> >
> >
> > I’ve got a custom registration class for Kryo which I think is working
> > - before I added this the original .map did not work – but shouldn’t
> > it be able to serialize all instances of Joda DateTime?
> >
> >
> >
> > Thank you for any help / pointers you can give me.
> >
> >
Re: NPE when using Joda DateTime
That's right, though it's possible the default way Kryo chooses to
serialize the object doesn't work. I'd debug a little more and print
out as much as you can about the DateTime object at the point it
appears to not work. I think there's a real problem and it only
happens to not turn up for the map + take(1) for reasons below.
Sandy I know you work with DateTime for spark-timeseries; does this ring a bell?
On Thu, Jan 14, 2016 at 2:28 PM, Spencer, Alex (Santander)
wrote:
> Hi,
>
> I tried take(1500) and test.collect and these both work on the "single" map
> statement.
>
> I'm very new to Kryo serialisation, I managed to find some code and I copied
> and pasted and that's what originally made the single map statement work:
>
> class MyRegistrator extends KryoRegistrator {
> override def registerClasses(kryo: Kryo) {
> kryo.register(classOf[org.joda.time.DateTime])
> }
> }
>
> Is it because the groupBy sees a different class type? Maybe Array[DateTime]?
> I don’t want to find the answer by trial and error though.
>
> Alex
>
> -Original Message-
> From: Sean Owen [mailto:so...@cloudera.com]
> Sent: 14 January 2016 14:07
> To: Spencer, Alex (Santander)
> Cc: user@spark.apache.org
> Subject: Re: NPE when using Joda DateTime
>
> It does look somehow like the state of the DateTime object isn't being
> recreated properly on deserialization somehow, given where the NPE occurs
> (look at the Joda source code). However the object is java.io.Serializable.
> Are you sure the Kryo serialization is correct?
>
> It doesn't quite explain why the map operation works by itself. It could be
> the difference between executing locally (take(1) will look at 1 partition in
> 1 task which prefers to be local) and executing remotely (groupBy is going to
> need a shuffle).
>
> On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander)
> wrote:
>> Hello,
>>
>>
>>
>> I was wondering if somebody is able to help me get to the bottom of a
>> null pointer exception I’m seeing in my code. I’ve managed to narrow
>> down a problem in a larger class to my use of Joda’s DateTime
>> functions. I’ve successfully run my code in scala, but I’ve hit a few
>> problems when adapting it to run in spark.
>>
>>
>>
>> Spark version: 1.3.0
>>
>> Scala version: 2.10.4
>>
>> Java HotSpot 1.7
>>
>>
>>
>> I have a small case class called Transaction, which looks something
>> like
>> this:
>>
>>
>>
>> case class Transaction(date : org.joda.time.DateTime = new
>> org.joda.time.DateTime())
>>
>>
>>
>> I have an RDD[Transactions] trans:
>>
>> org.apache.spark.rdd.RDD[Transaction] = MapPartitionsRDD[4] at map at
>> :44
>>
>>
>>
>> I am able to run this successfully:
>>
>>
>>
>> val test = trans.map(_.date.minusYears(10))
>>
>> test.take(1)
>>
>>
>>
>> However if I do:
>>
>>
>>
>> val groupedTrans = trans.groupBy(_.account)
>>
>>
>>
>> //For each group, process transactions in turn:
>>
>> val test = groupedTrans.flatMap { case (_, transList) =>
>>
>> transList.map {transaction =>
>>
>> transaction.date.minusYears(10)
>>
>> }
>>
>> }
>>
>> test.take(1)
>>
>>
>>
>> I get:
>>
>>
>>
>> java.lang.NullPointerException
>>
>> at org.joda.time.DateTime.minusYears(DateTime.java:1268)
>>
>>
>>
>> Should the second operation not be equivalent to the first .map one?
>> (It’s a long way round of producing my error – but it’s extremely
>> similar to what’s happening in my class).
>>
>>
>>
>> I’ve got a custom registration class for Kryo which I think is working
>> - before I added this the original .map did not work – but shouldn’t
>> it be able to serialize all instances of Joda DateTime?
>>
>>
>>
>> Thank you for any help / pointers you can give me.
>>
>>
>>
>> Kind Regards,
>>
>> Alex.
>>
>>
>>
>> Alex Spencer
>>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional
> commands, e-mail: user-h...@spark.apache.org
>
> Emails aren't always secure, and they may be intercepted or changed after
> they've been sent. Santander doesn't accept liability if this happens. If you
> think someone may have inter
RE: NPE when using Joda DateTime
Hi,
I tried take(1500) and test.collect and these both work on the "single" map
statement.
I'm very new to Kryo serialisation, I managed to find some code and I copied
and pasted and that's what originally made the single map statement work:
class MyRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[org.joda.time.DateTime])
}
}
Is it because the groupBy sees a different class type? Maybe Array[DateTime]? I
don’t want to find the answer by trial and error though.
Alex
-Original Message-
From: Sean Owen [mailto:so...@cloudera.com]
Sent: 14 January 2016 14:07
To: Spencer, Alex (Santander)
Cc: user@spark.apache.org
Subject: Re: NPE when using Joda DateTime
It does look somehow like the state of the DateTime object isn't being
recreated properly on deserialization somehow, given where the NPE occurs (look
at the Joda source code). However the object is java.io.Serializable. Are you
sure the Kryo serialization is correct?
It doesn't quite explain why the map operation works by itself. It could be the
difference between executing locally (take(1) will look at 1 partition in 1
task which prefers to be local) and executing remotely (groupBy is going to
need a shuffle).
On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander)
wrote:
> Hello,
>
>
>
> I was wondering if somebody is able to help me get to the bottom of a
> null pointer exception I’m seeing in my code. I’ve managed to narrow
> down a problem in a larger class to my use of Joda’s DateTime
> functions. I’ve successfully run my code in scala, but I’ve hit a few
> problems when adapting it to run in spark.
>
>
>
> Spark version: 1.3.0
>
> Scala version: 2.10.4
>
> Java HotSpot 1.7
>
>
>
> I have a small case class called Transaction, which looks something
> like
> this:
>
>
>
> case class Transaction(date : org.joda.time.DateTime = new
> org.joda.time.DateTime())
>
>
>
> I have an RDD[Transactions] trans:
>
> org.apache.spark.rdd.RDD[Transaction] = MapPartitionsRDD[4] at map at
> :44
>
>
>
> I am able to run this successfully:
>
>
>
> val test = trans.map(_.date.minusYears(10))
>
> test.take(1)
>
>
>
> However if I do:
>
>
>
> val groupedTrans = trans.groupBy(_.account)
>
>
>
> //For each group, process transactions in turn:
>
> val test = groupedTrans.flatMap { case (_, transList) =>
>
> transList.map {transaction =>
>
> transaction.date.minusYears(10)
>
> }
>
> }
>
> test.take(1)
>
>
>
> I get:
>
>
>
> java.lang.NullPointerException
>
> at org.joda.time.DateTime.minusYears(DateTime.java:1268)
>
>
>
> Should the second operation not be equivalent to the first .map one?
> (It’s a long way round of producing my error – but it’s extremely
> similar to what’s happening in my class).
>
>
>
> I’ve got a custom registration class for Kryo which I think is working
> - before I added this the original .map did not work – but shouldn’t
> it be able to serialize all instances of Joda DateTime?
>
>
>
> Thank you for any help / pointers you can give me.
>
>
>
> Kind Regards,
>
> Alex.
>
>
>
> Alex Spencer
>
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional
commands, e-mail: user-h...@spark.apache.org
Emails aren't always secure, and they may be intercepted or changed after
they've been sent. Santander doesn't accept liability if this happens. If you
think someone may have interfered with this email, please get in touch with the
sender another way. This message doesn't create or change any contract.
Santander doesn't accept responsibility for damage caused by any viruses
contained in this email or its attachments. Emails may be monitored. If you've
received this email by mistake, please let the sender know at once that it's
gone to the wrong person and then destroy it without copying, using, or telling
anyone about its contents.
Santander UK plc Reg. No. 2294747 and Abbey National Treasury Services plc Reg.
No. 2338548 Registered Offices: 2 Triton Square, Regent's Place, London NW1 3AN.
Registered in England. www.santander.co.uk. Authorised by the Prudential
Regulation Authority and regulated by the Financial Conduct Authority and the
Prudential Regulation Authority. FCA Reg. No. 106054 and 146003 respectively.
Santander Sharedealing is a trading name of Abbey Stockbrokers Limited Reg. No.
02666793. Registered Office: Kingfisher House, Radford Way, Billericay, Essex
CM12 0GZ. Authorised and regulated by the Financial Conduct Authority. FCA Reg.
No. 154210. You can chec
Re: NPE when using Joda DateTime
It does look somehow like the state of the DateTime object isn't being
recreated properly on deserialization somehow, given where the NPE
occurs (look at the Joda source code). However the object is
java.io.Serializable. Are you sure the Kryo serialization is correct?
It doesn't quite explain why the map operation works by itself. It
could be the difference between executing locally (take(1) will look
at 1 partition in 1 task which prefers to be local) and executing
remotely (groupBy is going to need a shuffle).
On Thu, Jan 14, 2016 at 1:01 PM, Spencer, Alex (Santander)
wrote:
> Hello,
>
>
>
> I was wondering if somebody is able to help me get to the bottom of a null
> pointer exception I’m seeing in my code. I’ve managed to narrow down a
> problem in a larger class to my use of Joda’s DateTime functions. I’ve
> successfully run my code in scala, but I’ve hit a few problems when adapting
> it to run in spark.
>
>
>
> Spark version: 1.3.0
>
> Scala version: 2.10.4
>
> Java HotSpot 1.7
>
>
>
> I have a small case class called Transaction, which looks something like
> this:
>
>
>
> case class Transaction(date : org.joda.time.DateTime = new
> org.joda.time.DateTime())
>
>
>
> I have an RDD[Transactions] trans:
>
> org.apache.spark.rdd.RDD[Transaction] = MapPartitionsRDD[4] at map at
> :44
>
>
>
> I am able to run this successfully:
>
>
>
> val test = trans.map(_.date.minusYears(10))
>
> test.take(1)
>
>
>
> However if I do:
>
>
>
> val groupedTrans = trans.groupBy(_.account)
>
>
>
> //For each group, process transactions in turn:
>
> val test = groupedTrans.flatMap { case (_, transList) =>
>
> transList.map {transaction =>
>
> transaction.date.minusYears(10)
>
> }
>
> }
>
> test.take(1)
>
>
>
> I get:
>
>
>
> java.lang.NullPointerException
>
> at org.joda.time.DateTime.minusYears(DateTime.java:1268)
>
>
>
> Should the second operation not be equivalent to the first .map one? (It’s a
> long way round of producing my error – but it’s extremely similar to what’s
> happening in my class).
>
>
>
> I’ve got a custom registration class for Kryo which I think is working -
> before I added this the original .map did not work – but shouldn’t it be
> able to serialize all instances of Joda DateTime?
>
>
>
> Thank you for any help / pointers you can give me.
>
>
>
> Kind Regards,
>
> Alex.
>
>
>
> Alex Spencer
>
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
NPE when using Joda DateTime
Hello,
I was wondering if somebody is able to help me get to the bottom of a null
pointer exception I'm seeing in my code. I've managed to narrow down a problem
in a larger class to my use of Joda's DateTime functions. I've successfully run
my code in scala, but I've hit a few problems when adapting it to run in spark.
Spark version: 1.3.0
Scala version: 2.10.4
Java HotSpot 1.7
I have a small case class called Transaction, which looks something like this:
case class Transaction(date : org.joda.time.DateTime = new
org.joda.time.DateTime())
I have an RDD[Transactions] trans:
org.apache.spark.rdd.RDD[Transaction] = MapPartitionsRDD[4] at map at
:44
I am able to run this successfully:
val test = trans.map(_.date.minusYears(10))
test.take(1)
However if I do:
val groupedTrans = trans.groupBy(_.account)
//For each group, process transactions in turn:
val test = groupedTrans.flatMap { case (_, transList) =>
transList.map {transaction =>
transaction.date.minusYears(10)
}
}
test.take(1)
I get:
java.lang.NullPointerException
at org.joda.time.DateTime.minusYears(DateTime.java:1268)
Should the second operation not be equivalent to the first .map one? (It's a
long way round of producing my error - but it's extremely similar to what's
happening in my class).
I've got a custom registration class for Kryo which I think is working - before
I added this the original .map did not work - but shouldn't it be able to
serialize all instances of Joda DateTime?
Thank you for any help / pointers you can give me.
Kind Regards,
Alex.
Alex Spencer
Emails aren't always secure, and they may be intercepted or changed after
they've been sent. Santander doesn't accept liability if this happens. If you
think someone may have interfered with this email, please get in touch with the
sender another way. This message doesn't create or change any contract.
Santander doesn't accept responsibility for damage caused by any viruses
contained in this email or its attachments. Emails may be monitored. If you've
received this email by mistake, please let the sender know at once that it's
gone to the wrong person and then destroy it without copying, using, or telling
anyone about its contents.
Santander UK plc Reg. No. 2294747 and Abbey National Treasury Services plc Reg.
No. 2338548 Registered Offices: 2 Triton Square, Regent's Place, London NW1 3AN.
Registered in England. www.santander.co.uk. Authorised by the Prudential
Regulation Authority and regulated by the Financial Conduct Authority and the
Prudential Regulation Authority. FCA Reg. No. 106054 and 146003 respectively.
Santander Sharedealing is a trading name of Abbey Stockbrokers Limited Reg. No.
02666793. Registered Office: Kingfisher House, Radford Way, Billericay, Essex
CM12 0GZ. Authorised and regulated by the Financial Conduct Authority. FCA Reg.
No. 154210. You can check this on the Financial Services Register by visiting
the FCA’s website www.fca.org.uk/register or by contacting the FCA on 0800 111
6768. Santander UK plc is also licensed by the Financial Supervision Commission
of the Isle of Man for its branch in the Isle of Man. Deposits held with the
Isle of Man branch are covered by the Isle of Man Depositors’ Compensation
Scheme as set out in the Isle of Man Depositors’ Compensation Scheme Regulations
2010. In the Isle of Man, Santander UK plc’s principal place of business is at
19/21 Prospect Hill, Douglas, Isle of Man, IM1 1ET. Santander and the flame logo
are registered trademarks.
Santander Asset Finance plc. Reg. No. 1533123. Registered Office: 2 Triton
Square, Regent’s Place, London NW1 3AN. Registered in England. Santander
Corporate & Commercial is a brand name used by Santander UK plc, Abbey National
Treasury Services plc and Santander Asset Finance plc.
Ref:[PDB#1-4A]
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
