Github user dtolpin commented on the pull request:
https://github.com/apache/spark/pull/9888#issuecomment-159662951
Any update on this please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user dtolpin opened a pull request:
https://github.com/apache/spark/pull/9888
[SPARK-11904] [PySpark] reduceByKeyAndWindow does not require checkpointing
when invFunc is None
when invFunc is None, `reduceByKeyAndWindow(func, None, winsize,
slidesize)` is equivalent to
Github user dtolpin closed the pull request at:
https://github.com/apache/spark/pull/9887
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user dtolpin commented on the pull request:
https://github.com/apache/spark/pull/9887#issuecomment-158689771
I did something wrong with rebasing, will remove and redo.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
GitHub user dtolpin opened a pull request:
https://github.com/apache/spark/pull/9887
[SPARK-11904] [PySpark] reduceByKeyAndWindow does not require checkpointing
when invFunc is None
when invFunc is None, `reduceByKeyAndWindow(func, None, winsize,
slidesize)` is equivalent to
Github user dtolpin commented on the pull request:
https://github.com/apache/spark/pull/9775#issuecomment-158032445
Got it, added it, thank you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user dtolpin commented on the pull request:
https://github.com/apache/spark/pull/9775#issuecomment-157873176
Sure, I have just put the summary back into the PR title.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user dtolpin commented on the pull request:
https://github.com/apache/spark/pull/9775#issuecomment-157806025
I don't see how I can make a meaningful unit test for this bug,
unfortunately. That would require that I use a time-based window with
reduceByKeyAndWindow, and fi
GitHub user dtolpin opened a pull request:
https://github.com/apache/spark/pull/9775
invFunc=none work properly with python's reduceByKeyAndWindow
invFunc is optional and can be None. Instead of invFunc (the parameter)
invReduceFunc (a local function) was checked for trueness
