[
https://issues.apache.org/jira/browse/BIGTOP-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264620#comment-14264620
]
RJ Nowling commented on BIGTOP-1586:
------------------------------------
Hi [~jayunit100],
The Calendar objects in Java are a PITA. They contain many fields, which are
populated via the current system properties. There is no way to override them
in the constructor but you can set them through the API. And not all of these
fields are written out when you convert the Calendar to a String to be able to
write it out. For example, in the unit tests, I set the millisecond to 0
because the Calendar object will use the milliseconds of the current system
time and there are no milliseconds values for the time data in the BigPetStore
output, which is generated by Calendar's own `toString` method.
This is further complicated by how I wrote the unit test. I do a simple
equality comparison on the case classes, which does an equality comparison on
the Calendar objects.
I think we need to clean up the unit tests first to make debugging easier than
figure out the correct solution. For the unit tests, we should compare every
field individually instead of the Transaction objects themselves. E.g., does
`transaction1.store == transaction2.store`, etc. For the date/time, we should
not directly compare the Calendar objects but just the day, month, year, hour,
minute, second, and time zone fields. The transaction comparisons should be
wrapped in a utility method so we can call it from other unit tests and
standardize it across the test suite.
Once that's done, we can figure out where the problem is with date/time parsing.
What do you think?
> BigPetStore-Spark only works on the East Coast .
> ------------------------------------------------
>
> Key: BIGTOP-1586
> URL: https://issues.apache.org/jira/browse/BIGTOP-1586
> Project: Bigtop
> Issue Type: Bug
> Components: blueprints
> Affects Versions: 0.9.0
> Reporter: jay vyas
>
> Yup, its true. i think :)
> When visiting my parents in *oklahoma* I found that the way bigpetstore-spark
> is set up, only people on the *right* coast can run the unit tests...
> something with the default time zone setup in java and the unit tests which
> test for set equivalence.
> {noformat}
> s
> Failed to match [Lscala.Tuple5;@7ac2933e and [Lscala.Tuple5;@7c510a68
> missing
> (Store(5,11553),Location(11553,Uniondale,NY),Customer(999,Cesareo,Lamplough,20152),Location(20152,Chantilly,VA),TransactionProduct(999,32,5,java.util.GregorianCalendar[time=1446530891000,areFieldsSet=true,areAllFieldsSet=false,lenient=true,zone=sun.util.calendar.ZoneInfo[id="America/Chicago",offset=-21600000,dstSavings=3600000,useDaylight=true,transitions=235,lastRule=java.util.SimpleTimeZone[id=America/Chicago,offset=-21600000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2015,MONTH=10,WEEK_OF_YEAR=45,WEEK_OF_MONTH=1,DAY_OF_MONTH=3,DAY_OF_YEAR=307,DAY_OF_WEEK=3,DAY_OF_WEEK_IN_MONTH=1,AM_PM=0,HOUR=0,HOUR_OF_DAY=0,MINUTE=8,SECOND=11,MILLISECOND=0,ZONE_OFFSET=-21600000,DST_OFFSET=0],category=dry
> dog food;brand=Happy Pup;flavor=Fish & Potato;size=30.0;per_unit_cost=2.67;))
> ... not found=
> (Store(5,11553),Location(11553,Uniondale,NY),Customer(999,Cesareo,Lamplough,20152),Location(20152,Chantilly,VA),TransactionProduct(999,32,5,java.util.GregorianCalendar[time=?,areFieldsSet=false,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2015,MONTH=9,WEEK_OF_YEAR=2,WEEK_OF_MONTH=2,DAY_OF_MONTH=12,DAY_OF_YEAR=5,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=1,AM_PM=0,HOUR=6,HOUR_OF_DAY=4,MINUTE=29,SECOND=46,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0],category=dry
> dog food;brand=Happy Pup;flavor=Fish & Potato;size=30.0;per_unit_cost=2.67;))
> ... not found= (Store(1,98110),Location(98110,Bainbridge
> Islan,WA),Customer(999,Cesareo,Lamplough,20152),Location(20152,Chantilly,VA),TransactionProduct(999,31,1,java.util.GregorianCalendar[time=?,areFieldsSet=false,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2015,MONTH=10,WEEK_OF_YEAR=2,WEEK_OF_MONTH=2,DAY_OF_MONTH=3,DAY_OF_YEAR=5,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=1,AM_PM=0,HOUR=6,HOUR_OF_DAY=1,MINUTE=8,SECOND=11,MILLISECOND=0,ZONE_OFFSET=0,DST_OFFSET=0],category=poop
> bags;brand=Dog Days;color=Blue;size=60.0;per_unit_cost=0.21;))
> ... not found=
> (Store(6,66067),Location(66067,Ottawa,KS),Customer(999,Cesareo,Lamplough,20152),Location(20152,Chantilly,VA),TransactionProduct
> {noformat}
> I've got a patch coming in which has some more general improvements to it,
> and in also some removal of the monads in the unit tests to make it easier to
> debug/run.
> I'll submit it shortly once i fully fix the time zone issue.
> Even though i love the east coast, I have a dream that anyone, regardless of
> the coast they are on, race, religion, or creed, can use spark to generate
> petabytes of fake data !
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
