[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813880#comment-16813880
]
shane knapp edited comment on SPARK-27389 at 4/9/19 10:54 PM:
--
btw, the total impact of this problem "only" failed 73 builds over the past
seven days and was limited to two workers, amp-jenkins-worker-03 and -05.
{noformat}
1 NewSparkPullRequestBuilder
2 spark-branch-2.3-test-sbt-hadoop-2.6
3 spark-branch-2.3-test-sbt-hadoop-2.7
1 spark-branch-2.4-test-sbt-hadoop-2.6
6 spark-branch-2.4-test-sbt-hadoop-2.7
11 spark-master-test-sbt-hadoop-2.7
49 SparkPullRequestBuilder
{noformat}
i still haven't figured out *why* things broke... it wasn't an errant package
install by a build as i have the anaconda dirs locked down and the only way to
add/update packages there is to use sudo.
was (Author: shaneknapp):
btw, the total impact of this problem "only" failed 73 builds over the past
seven days and was limited to two workers, amp-jenkins-worker-03 and -05.
i still haven't figured out *why* things broke... it wasn't an errant package
install by a build as i have the anaconda dirs locked down and the only way to
add/update packages there is to use sudo.
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
> (pandas/tslib.c:32362)
> File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line
> 178, in timezone
> raise UnknownTimeZoneError(zone)
> UnknownTimeZoneError: 'US/Pacific-New'
> {noformat}
--
This message was
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813812#comment-16813812
]
shane knapp edited comment on SPARK-27389 at 4/9/19 9:09 PM:
-
ok, this should be fixed now... i got all the workers to recognize
US/Pacific-New w/python2.7 and the python/run-tests script now passes!
the following was run on amp-jenkins-worker-05, which was failing continuously
w/the unknown tz error:
{noformat}
-bash-4.1$ python/run-tests --python-executables=python2.7
Running PySpark tests. Output is in
/home/jenkins/src/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml',
'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming']
...trimming a bunch to get to the failing tests...
Finished test(python2.7): pyspark.sql.tests.test_dataframe (32s) ... 2 tests
were skipped
...yay! it passed! now skipping more output to get to the end...
Tests passed in 797 seconds
Skipped tests in pyspark.sql.tests.test_dataframe with python2.7:
test_create_dataframe_required_pandas_not_found
(pyspark.sql.tests.test_dataframe.DataFrameTests) ... skipped 'Required Pandas
was found.'
test_to_pandas_required_pandas_not_found
(pyspark.sql.tests.test_dataframe.DataFrameTests) ... skipped 'Required Pandas
was found.'
{noformat}
turns out that a couple of workers were missing the US/Pacific-New tzinfo file
in the pytz libdir. a quick scp + python2.7 -m compileall later and things
seem to be happy!
i'll leave this open for now, and if anyone notices other builds failing in
this way please link to them here.
was (Author: shaneknapp):
ok, this should be fixed now... i got all the workers to recognize
US/Pacific-New w/python2.7 and the python/run-tests script now passes!
{noformat}
-bash-4.1$ python/run-tests --python-executables=python2.7
Running PySpark tests. Output is in
/home/jenkins/src/spark/python/unit-tests.log
Will test against the following Python executables: ['python2.7']
Will test the following Python modules: ['pyspark-core', 'pyspark-ml',
'pyspark-mllib', 'pyspark-sql', 'pyspark-streaming']
...trimming a bunch to get to the failing tests...
Finished test(python2.7): pyspark.sql.tests.test_dataframe (32s) ... 2 tests
were skipped
...yay! it passed! now skipping more output to get to the end...
Tests passed in 797 seconds
Skipped tests in pyspark.sql.tests.test_dataframe with python2.7:
test_create_dataframe_required_pandas_not_found
(pyspark.sql.tests.test_dataframe.DataFrameTests) ... skipped 'Required Pandas
was found.'
test_to_pandas_required_pandas_not_found
(pyspark.sql.tests.test_dataframe.DataFrameTests) ... skipped 'Required Pandas
was found.'
{noformat}
turns out that a couple of workers were missing the US/Pacific-New tzinfo file
in the pytz libdir. a quick scp + python2.7 -m compileall later and things
seem to be happy!
i'll leave this open for now, and if anyone notices other builds failing in
this way please link to them here.
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
>
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813625#comment-16813625
]
shane knapp edited comment on SPARK-27389 at 4/9/19 5:08 PM:
-
done.
{noformat}
$ pssh -h jenkins_workers.txt "cp /root/python/__init__.py
/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py"
[1] 10:06:19 [SUCCESS] amp-jenkins-worker-02
[2] 10:06:19 [SUCCESS] amp-jenkins-worker-06
[3] 10:06:19 [SUCCESS] amp-jenkins-worker-05
[4] 10:06:19 [SUCCESS] amp-jenkins-worker-03
[5] 10:06:19 [SUCCESS] amp-jenkins-worker-01
[6] 10:06:19 [SUCCESS] amp-jenkins-worker-04
$ ssh amp-jenkins-worker-03 "grep Pacific-New
/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py"
'US/Pacific-New',
'US/Pacific-New',
{noformat}
and
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ python2.7 -c "import pytz; print
'US/Pacific-New' in pytz.all_timezones"
True
{noformat}
was (Author: shaneknapp):
done.
{noformat}
$ pssh -h jenkins_workers.txt "cp /root/python/__init__.py
/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py"
[1] 10:06:19 [SUCCESS] amp-jenkins-worker-02
[2] 10:06:19 [SUCCESS] amp-jenkins-worker-06
[3] 10:06:19 [SUCCESS] amp-jenkins-worker-05
[4] 10:06:19 [SUCCESS] amp-jenkins-worker-03
[5] 10:06:19 [SUCCESS] amp-jenkins-worker-01
[6] 10:06:19 [SUCCESS] amp-jenkins-worker-04
$ ssh amp-jenkins-worker-03 "grep Pacific-New
/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py"
'US/Pacific-New',
'US/Pacific-New',
{noformat}
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812883#comment-16812883
]
shane knapp edited comment on SPARK-27389 at 4/9/19 12:21 AM:
--
-no, it appears to be random.-
-[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/365/]-
-[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/364/]-
-these two identical builds ran w/the same python/java/whathaveyou setup on the
*same physical worker*. one passes, one fails w/the date thing.-
bad example, pls hold.
i need to do some more build archaeology this evening and tomorrow. i'm aware
that this is important. :)
was (Author: shaneknapp):
-no, it appears to be random.-
-[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/365/]-
-[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/364/]-
-these two identical builds ran w/the same python/java/whathaveyou setup on the
*same physical worker*. one passes, one fails w/the date thing.-
bad example, pls hold.
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
> (pandas/tslib.c:32362)
> File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line
> 178, in timezone
> raise UnknownTimeZoneError(zone)
> UnknownTimeZoneError: 'US/Pacific-New'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812883#comment-16812883
]
shane knapp edited comment on SPARK-27389 at 4/9/19 12:05 AM:
--
-no, it appears to be random.-
-[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/365/]-
-[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/364/]-
-these two identical builds ran w/the same python/java/whathaveyou setup on the
*same physical worker*. one passes, one fails w/the date thing.-
bad example, pls hold.
was (Author: shaneknapp):
no, it appears to be random.
[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/365/]
[https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.4-test-sbt-hadoop-2.7/364/]
these two identical builds ran w/the same python/java/whathaveyou setup on the
*same physical worker*. one passes, one fails w/the date thing.
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
> (pandas/tslib.c:32362)
> File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line
> 178, in timezone
> raise UnknownTimeZoneError(zone)
> UnknownTimeZoneError: 'US/Pacific-New'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812841#comment-16812841
]
shane knapp edited comment on SPARK-27389 at 4/8/19 10:07 PM:
--
also, java8 appears to believe i'm in the US/Pacific (not Pacific-New) TZ:
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ cat tz.java
import java.util.TimeZone;
public class tz {
public static void main(String[] args) {
TimeZone tz = TimeZone.getDefault();
System.out.println(tz.getID());
}
}
[sknapp@amp-jenkins-worker-04 ~]$ javac tz.java
[sknapp@amp-jenkins-worker-04 ~]$ java tz
US/Pacific
[sknapp@amp-jenkins-worker-04 ~]$ java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
{noformat}
was (Author: shaneknapp):
also, java8 appears to believe i'm in the US/Pacific (not Pacific-New) TZ:
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ cat tz.java
import java.util.TimeZone;
public class tz {
public static void main(String[] args) {
TimeZone tz = TimeZone.getDefault();
System.out.println(tz.getID());
}
}
[sknapp@amp-jenkins-worker-04 ~]$ javac tz.java
[sknapp@amp-jenkins-worker-04 ~]$ java tz
US/Pacific
[sknapp@amp-jenkins-worker-04 ~]$ java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
{noormat}
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
>
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812841#comment-16812841
]
shane knapp edited comment on SPARK-27389 at 4/8/19 10:06 PM:
--
also, java8 appears to believe i'm in the US/Pacific (not Pacific-New) TZ:
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ cat tz.java
import java.util.TimeZone;
public class tz {
public static void main(String[] args) {
TimeZone tz = TimeZone.getDefault();
System.out.println(tz.getID());
}
}
[sknapp@amp-jenkins-worker-04 ~]$ javac tz.java
[sknapp@amp-jenkins-worker-04 ~]$ java tz
US/Pacific
[sknapp@amp-jenkins-worker-04 ~]$ java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
{noormat}
was (Author: shaneknapp):
also, java8 appears to believe i'm in the US/Pacific (not Pacific-New) TZ:
{preformat}
[sknapp@amp-jenkins-worker-04 ~]$ cat tz.java
import java.util.TimeZone;
public class tz {
public static void main(String[] args) {
TimeZone tz = TimeZone.getDefault();
System.out.println(tz.getID());
}
}
[sknapp@amp-jenkins-worker-04 ~]$ javac tz.java
[sknapp@amp-jenkins-worker-04 ~]$ java tz
US/Pacific
[sknapp@amp-jenkins-worker-04 ~]$ java -version
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
{preformat}
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
>
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812789#comment-16812789
]
shane knapp edited comment on SPARK-27389 at 4/8/19 9:09 PM:
-
updating tzdata (via pip and yum) didn't do anything noticeable:
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ python2.7 -c 'import pytz; print
"US/Pacific-New" in pytz.all_timezones'
False
[sknapp@amp-jenkins-worker-04 ~]$ which python2.7
/home/anaconda/bin/python2.7
{noformat}
this is actually expected as pytz stores it's OWN tzdata (see my earlier
comment about hacking anaconda/lib/python2.7/site-packages/pytz/__init__.py).
was (Author: shaneknapp):
updating tzdata didn't do anything noticeable:
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ python2.7 -c 'import pytz; print
"US/Pacific-New" in pytz.all_timezones'
False
[sknapp@amp-jenkins-worker-04 ~]$ which python2.7
/home/anaconda/bin/python2.7
{noformat}
this is actually expected as pytz stores it's OWN tzdata (see my earlier
comment about hacking anaconda/lib/python2.7/site-packages/pytz/__init__.py).
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
> (pandas/tslib.c:32362)
> File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line
> 178, in timezone
> raise UnknownTimeZoneError(zone)
> UnknownTimeZoneError: 'US/Pacific-New'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812789#comment-16812789
]
shane knapp edited comment on SPARK-27389 at 4/8/19 9:03 PM:
-
updating tzdata didn't do anything noticeable:
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ python2.7 -c 'import pytz; print
"US/Pacific-New" in pytz.all_timezones'
False
[sknapp@amp-jenkins-worker-04 ~]$ which python2.7
/home/anaconda/bin/python2.7
{noformat}
this is actually expected as pytz stores it's OWN tzdata (see my earlier
comment about hacking anaconda/lib/python2.7/site-packages/pytz/__init__.py).
was (Author: shaneknapp):
updating tzdata didn't do anything noticeable:
{noformat}
[sknapp@amp-jenkins-worker-04 ~]$ python2.7 -c 'import pytz; print
"US/Pacific-New" in pytz.all_timezones'
False
[sknapp@amp-jenkins-worker-04 ~]$ which python2.7
/home/anaconda/bin/python2.7
{noformat}
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1878, in
> if ts is not pd.NaT else pd.NaT)
> File "pandas/tslib.pyx", line 649, in pandas.tslib.Timestamp.tz_convert
> (pandas/tslib.c:13923)
> File "pandas/tslib.pyx", line 407, in pandas.tslib.Timestamp.__new__
> (pandas/tslib.c:10447)
> File "pandas/tslib.pyx", line 1467, in pandas.tslib.convert_to_tsobject
> (pandas/tslib.c:27504)
> File "pandas/tslib.pyx", line 1768, in pandas.tslib.maybe_get_tz
> (pandas/tslib.c:32362)
> File "/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py", line
> 178, in timezone
> raise UnknownTimeZoneError(zone)
> UnknownTimeZoneError: 'US/Pacific-New'
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27389) pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
[
https://issues.apache.org/jira/browse/SPARK-27389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812704#comment-16812704
]
shane knapp edited comment on SPARK-27389 at 4/8/19 6:56 PM:
-
is this even really a valid timezone?
plus, i really don't think this is a jenkins issue per se. i whipped up some
java to check for this timezone, which is there:
{code}
$ java DisplayZoneAndOffSet|grep Pacific-New
US/Pacific-New (UTC-07:00)
{code}
but it's definitely not a valid pytz timezone:
{code}
$ python2.7 -c 'import pytz; print "US/Pacific-New" in pytz.all_timezones'
False
{code}
we're also running the latest version of pytz (according to pip at least):
{code}
$ pip2.7 install -U pytz
Requirement already up-to-date: pytz in
/home/anaconda/lib/python2.7/site-packages (2018.9)
$ pip2.7 show pytz
Name: pytz
Version: 2018.9
Summary: World timezone definitions, modern and historical
Home-page: http://pythonhosted.org/pytz
Author: Stuart Bishop
Author-email: stu...@stuartbishop.net
License: MIT
Location: /home/anaconda/lib/python2.7/site-packages
Requires:
Required-by: pandas
{code}
as a work-around... i *could* hack
{code}/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py{code} to
include US/Pacific-New on all of the workers. ;)
was (Author: shaneknapp):
is this even really a valid timezone?
plus, i really don't think this is a jenkins issue per se. i whipped up some
java to check for this timezone, which is there:
{code}
$ java DisplayZoneAndOffSet|grep Pacific-New
US/Pacific-New (UTC-07:00)
{code}
but it's definitely not a valid pytz timezone:
{code}
$ python2.7 -c 'import pytz; print "US/Pacific-New" in pytz.all_timezones'
False
{code}
as a work-around... i *could* hack
{code}/home/anaconda/lib/python2.7/site-packages/pytz/__init__.py{code} to
include US/Pacific-New on all of the workers. ;)
> pyspark test failures w/ "UnknownTimeZoneError: 'US/Pacific-New'"
> -
>
> Key: SPARK-27389
> URL: https://issues.apache.org/jira/browse/SPARK-27389
> Project: Spark
> Issue Type: Task
> Components: jenkins, PySpark
>Affects Versions: 3.0.0
>Reporter: Imran Rashid
>Assignee: shane knapp
>Priority: Major
>
> I've seen a few odd PR build failures w/ an error in pyspark tests about
> "UnknownTimeZoneError: 'US/Pacific-New'". eg.
> https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4688/consoleFull
> A bit of searching tells me that US/Pacific-New probably isn't really
> supposed to be a timezone at all:
> https://mm.icann.org/pipermail/tz/2009-February/015448.html
> I'm guessing that this is from some misconfiguration of jenkins. that said,
> I can't figure out what is wrong. There does seem to be a timezone entry for
> US/Pacific-New in {{/usr/share/zoneinfo/US/Pacific-New}} -- but it seems to
> be there on every amp-jenkins-worker, so I dunno what that alone would cause
> this failure sometime.
> [~shaneknapp] I am tentatively calling this a "jenkins" issue, but I might be
> totally wrong here and it is really a pyspark problem.
> Full Stack trace from the test failure:
> {noformat}
> ==
> ERROR: test_to_pandas (pyspark.sql.tests.test_dataframe.DataFrameTests)
> --
> Traceback (most recent call last):
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 522, in test_to_pandas
> pdf = self._to_pandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/tests/test_dataframe.py",
> line 517, in _to_pandas
> return df.toPandas()
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/dataframe.py",
> line 2189, in toPandas
> _check_series_convert_timestamps_local_tz(pdf[field.name], timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1891, in _check_series_convert_timestamps_local_tz
> return _check_series_convert_timestamps_localize(s, None, timezone)
> File
> "/home/jenkins/workspace/NewSparkPullRequestBuilder@2/python/pyspark/sql/types.py",
> line 1877, in _check_series_convert_timestamps_localize
> lambda ts: ts.tz_localize(from_tz,
> ambiguous=False).tz_convert(to_tz).tz_localize(None)
> File "/home/anaconda/lib/python2.7/site-packages/pandas/core/series.py",
> line 2294, in apply
> mapped = lib.map_infer(values, f, convert=convert_dtype)
> File "pandas/src/inference.pyx", line 1207, in pandas.lib.map_infer
> (pandas/lib.c:66124)
> File
>
