[
https://issues.apache.org/jira/browse/BEAM-9547?focusedWorklogId=613725&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-613725
]
ASF GitHub Bot logged work on BEAM-9547:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Jun/21 22:24
Start Date: 22/Jun/21 22:24
Worklog Time Spent: 10m
Work Description: TheNeuralBit commented on a change in pull request
#14940:
URL: https://github.com/apache/beam/pull/14940#discussion_r656613692
##########
File path: sdks/python/apache_beam/dataframe/frames_test.py
##########
@@ -891,6 +891,74 @@ def test_cat(self):
self._run_test(lambda s: s.cat.as_unordered(), s)
self._run_test(lambda s: s.cat.codes, s)
+ @parameterized.expand(frames.ELEMENTWISE_DATETIME_PROPERTIES)
+ def test_dt_property(self, prop_name):
+ # Generate a series with a lot of unique timestamps
+ s = pd.Series(
+ pd.date_range('1/1/2000', periods=100, freq='m') +
Review comment:
I don't think it matters. Some of the properties are ambiguous in this
case (example `.day` could be different depending on the timezone), but pandas
has made a decision about how to handle these ambiguities, and they allow it.
Here we're just verifying we do the same thing as pandas.
##########
File path: sdks/python/apache_beam/dataframe/frames.py
##########
@@ -4042,6 +4047,134 @@ def codes(self):
make_cat_func(method), name=method,
base=pd.core.arrays.categorical.CategoricalAccessor))
+class _DeferredDatetimeMethods(frame_base.DeferredBase):
+ @property # type: ignore
+ @frame_base.with_docs_from(pd.core.indexes.accessors.DatetimeProperties)
+ def tz(self):
+ return self._expr.proxy().dt.tz
+
+ @property # type: ignore
+ @frame_base.with_docs_from(pd.core.indexes.accessors.DatetimeProperties)
+ def freq(self):
+ return self._expr.proxy().dt.freq
+
+ @frame_base.with_docs_from(pd.core.indexes.accessors.DatetimeProperties)
+ def tz_localize(self, *args, ambiguous='infer', **kwargs):
+ """``ambiguous`` cannot be set to ``"infer"`` as its semantics are
+ order-sensitive. Similarly, specifying ``ambiguous`` as an
+ :class:`~numpy.ndarray` is order-sensitive, but you can achieve similar
+ functionality by specifying ``ambiguous`` as a Series."""
+ if isinstance(ambiguous, np.ndarray):
+ raise frame_base.WontImplementError(
+ "tz_localize(ambiguous=ndarray) is not supported because it makes "
+ "this operation sensitive to the order of the data. Please use a "
+ "DeferredSeries instead.",
+ reason="order-sensitive")
+ elif isinstance(ambiguous, frame_base.DeferredFrame):
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'tz_localize',
+ lambda s,
+ ambiguous: s.dt.tz_localize(*args, ambiguous=ambiguous,
**kwargs),
+ [self._expr, ambiguous._expr],
+ requires_partition_by=partitionings.Index(),
+ preserves_partition_by=partitionings.Arbitrary()))
+ elif ambiguous == 'infer':
+ # infer attempts to infer based on the order of the timestamps
+ raise frame_base.WontImplementError(
+ f"tz_localize(ambiguous={ambiguous!r}) is not allowed because it "
+ "makes this operation sensitive to the order of the data.",
+ reason="order-sensitive")
+
+ return frame_base.DeferredFrame.wrap(
+ expressions.ComputedExpression(
+ 'tz_localize',
+ lambda s: s.dt.tz_localize(*args, ambiguous=ambiguous, **kwargs),
+ [self._expr],
+ requires_partition_by=partitionings.Arbitrary(),
+ preserves_partition_by=partitionings.Arbitrary()))
+
+
+ to_period = frame_base.not_implemented_method(
Review comment:
Thanks - I just wasn't quite sure what to do with this so I punted on
it. I think we'll need event-time semantics integration to do it right, so I
changed it to WontImplement(event-time-semantics).
##########
File path: sdks/python/apache_beam/dataframe/pandas_doctests_test.py
##########
@@ -545,43 +543,46 @@ def test_string_tests(self):
def test_datetime_tests(self):
# TODO(BEAM-10721)
- datetimelike_result = doctests.testmod(
- pd.core.arrays.datetimelike,
+ indexes_accessors_result = doctests.testmod(
+ pd.core.indexes.accessors,
use_beam=False,
skip={
- 'pandas.core.arrays.datetimelike.AttributesMixin._unbox_scalar': [
+ 'pandas.core.indexes.accessors.TimedeltaProperties': [
+ # Seems like an upstream bug. The property is 'second'
+ 'seconds_series.dt.seconds'
+ ],
+
+ # Test data creation fails for these:
Review comment:
Thanks for asking about this. This is an issue with the way we shim
constant values in the doctest framework. I filed BEAM-12530 for this and
linked it here.
Also I had opened https://github.com/apache/beam/pull/13457 for this a while
ago and completely forgot about it. Probably worth dusting that off for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 613725)
Time Spent: 147h 20m (was: 147h 10m)
> Implement all pandas operations (or raise WontImplementError)
> -------------------------------------------------------------
>
> Key: BEAM-9547
> URL: https://issues.apache.org/jira/browse/BEAM-9547
> Project: Beam
> Issue Type: Improvement
> Components: dsl-dataframe, sdk-py-core
> Reporter: Brian Hulette
> Assignee: Robert Bradshaw
> Priority: P2
> Labels: dataframe-api
> Time Spent: 147h 20m
> Remaining Estimate: 0h
>
> We should have an implementation for every DataFrame, Series, and GroupBy
> method. Everything that's not possible to implement should get a default
> implementation that raises WontImplementError
> See https://github.com/apache/beam/pull/10757#discussion_r389132292
> Progress at the individual operation level is tracked in a
> [spreadsheet|https://docs.google.com/spreadsheets/d/1hHAaJ0n0k2tw465ORs5tfdy4Lg0DnGWIQ53cLjAhel0/edit],
> consider requesting edit access if you'd like to help out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
