ail: [hidden email]
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Maciej Szymkiewicz
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
node+[hidden
> > email]]
> > Sent: Wednesday, November 30, 2016 8:35 PM
> > To: Mendelson, Assaf
> > Subject: Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function
> > frame boundary API
> >
> > Yes I'd define unboundedPreceding to -sys.maxsize, but a
; Assaf.
>
>
>
> *From:*rxin [via Apache Spark Developers List] [mailto:ml-node+[hidden
> email] ]
> *Sent:* Wednesday, November 30, 2016 8:35 PM
> *To:* Mendelson, Assaf
> *Subject:* Re: [SPARK-17845] [SQL][PYTHON] More self-evident window
> function frame
It could be something like this
https://github.com/zero323/spark/commit/b1f4d8218629b56b0982ee58f5b93a40305985e0
but I am not fully satisfied.
On 11/30/2016 07:34 PM, Reynold Xin wrote:
> Yes I'd define unboundedPreceding to -sys.maxsize, but also any value
> less than min(-sys.maxsize,
: [SPARK-17845] [SQL][PYTHON] More self-evident window function
frame boundary API
Yes I'd define unboundedPreceding to -sys.maxsize, but also any value less than
min(-sys.maxsize, _JAVA_MIN_LONG) are considered unboundedPreceding too. We
need to be careful with long overflow when transferring data
Yes I'd define unboundedPreceding to -sys.maxsize, but also any value less
than min(-sys.maxsize, _JAVA_MIN_LONG) are considered unboundedPreceding
too. We need to be careful with long overflow when transferring data over
to Java.
On Wed, Nov 30, 2016 at 10:04 AM, Maciej Szymkiewicz
It is platform specific so theoretically can be larger, but 2**63 - 1 is
a standard on 64 bit platform and 2**31 - 1 on 32bit platform. I can
submit a patch but I am not sure how to proceed. Personally I would set
unboundedPreceding = -sys.maxsize
unboundedFollowing = sys.maxsize
to keep
Ah ok for some reason when I did the pull request sys.maxsize was much
larger than 2^63. Do you want to submit a patch to fix this?
On Wed, Nov 30, 2016 at 9:48 AM, Maciej Szymkiewicz
wrote:
> The problem is that -(1 << 63) is -(sys.maxsize + 1) so the code which
> used
The problem is that -(1 << 63) is -(sys.maxsize + 1) so the code which
used to work before is off by one.
On 11/30/2016 06:43 PM, Reynold Xin wrote:
> Can you give a repro? Anything less than -(1 << 63) is considered
> negative infinity (i.e. unbounded preceding).
>
> On Wed, Nov 30, 2016 at 8:27
Can you give a repro? Anything less than -(1 << 63) is considered negative
infinity (i.e. unbounded preceding).
On Wed, Nov 30, 2016 at 8:27 AM, Maciej Szymkiewicz
wrote:
> Hi,
>
> I've been looking at the SPARK-17845 and I am curious if there is any
> reason to make it
10 matches
Mail list logo