Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-12-02 Thread Maciej Szymkiewicz
ail: [hidden email] >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Maciej Szymkiewicz >>> >>> >>> >>> >>> >>> -- >>>

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-12-01 Thread Reynold Xin
node+[hidden > > email]] > > Sent: Wednesday, November 30, 2016 8:35 PM > > To: Mendelson, Assaf > > Subject: Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function > > frame boundary API > > > > Yes I'd define unboundedPreceding to -sys.maxsize, but a

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-12-01 Thread Maciej Szymkiewicz
; Assaf. > > > > *From:*rxin [via Apache Spark Developers List] [mailto:ml-node+[hidden > email] ] > *Sent:* Wednesday, November 30, 2016 8:35 PM > *To:* Mendelson, Assaf > *Subject:* Re: [SPARK-17845] [SQL][PYTHON] More self-evident window > function frame

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-12-01 Thread Maciej Szymkiewicz
It could be something like this https://github.com/zero323/spark/commit/b1f4d8218629b56b0982ee58f5b93a40305985e0 but I am not fully satisfied. On 11/30/2016 07:34 PM, Reynold Xin wrote: > Yes I'd define unboundedPreceding to -sys.maxsize, but also any value > less than min(-sys.maxsize,

RE: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-11-30 Thread assaf.mendelson
: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API Yes I'd define unboundedPreceding to -sys.maxsize, but also any value less than min(-sys.maxsize, _JAVA_MIN_LONG) are considered unboundedPreceding too. We need to be careful with long overflow when transferring data

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-11-30 Thread Reynold Xin
Yes I'd define unboundedPreceding to -sys.maxsize, but also any value less than min(-sys.maxsize, _JAVA_MIN_LONG) are considered unboundedPreceding too. We need to be careful with long overflow when transferring data over to Java. On Wed, Nov 30, 2016 at 10:04 AM, Maciej Szymkiewicz

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-11-30 Thread Maciej Szymkiewicz
It is platform specific so theoretically can be larger, but 2**63 - 1 is a standard on 64 bit platform and 2**31 - 1 on 32bit platform. I can submit a patch but I am not sure how to proceed. Personally I would set unboundedPreceding = -sys.maxsize unboundedFollowing = sys.maxsize to keep

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-11-30 Thread Reynold Xin
Ah ok for some reason when I did the pull request sys.maxsize was much larger than 2^63. Do you want to submit a patch to fix this? On Wed, Nov 30, 2016 at 9:48 AM, Maciej Szymkiewicz wrote: > The problem is that -(1 << 63) is -(sys.maxsize + 1) so the code which > used

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-11-30 Thread Maciej Szymkiewicz
The problem is that -(1 << 63) is -(sys.maxsize + 1) so the code which used to work before is off by one. On 11/30/2016 06:43 PM, Reynold Xin wrote: > Can you give a repro? Anything less than -(1 << 63) is considered > negative infinity (i.e. unbounded preceding). > > On Wed, Nov 30, 2016 at 8:27

Re: [SPARK-17845] [SQL][PYTHON] More self-evident window function frame boundary API

2016-11-30 Thread Reynold Xin
Can you give a repro? Anything less than -(1 << 63) is considered negative infinity (i.e. unbounded preceding). On Wed, Nov 30, 2016 at 8:27 AM, Maciej Szymkiewicz wrote: > Hi, > > I've been looking at the SPARK-17845 and I am curious if there is any > reason to make it