[GitHub] spark pull request #20817: [SPARK-23599][SQL] Add a UUID generator from Pseu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20817


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20817: [SPARK-23599][SQL] Add a UUID generator from Pseu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20817#discussion_r174995443
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RandomUUIDGenerator.scala
 ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import java.util.UUID
+
+import org.apache.commons.math3.random.MersenneTwister
+
+/**
+ * This class is used to generate a UUID from Pseudo-Random Numbers.
+ *
+ * For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) 
URN Namespace,
+ * section 4.4 "Algorithms for Creating a UUID from Truly Random or 
Pseudo-Random Numbers".
+ */
+case class RandomUUIDGenerator(randomSeed: Long) {
+  private val random = new MersenneTwister(randomSeed)
+
+  def getNextUUID(): UUID = {
--- End diff --

Sounds good. I've added it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20817: [SPARK-23599][SQL] Add a UUID generator from Pseu...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/20817#discussion_r174811371
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RandomUUIDGenerator.scala
 ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import java.util.UUID
+
+import org.apache.commons.math3.random.MersenneTwister
+
+/**
+ * This class is used to generate a UUID from Pseudo-Random Numbers.
+ *
+ * For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) 
URN Namespace,
+ * section 4.4 "Algorithms for Creating a UUID from Truly Random or 
Pseudo-Random Numbers".
+ */
+case class RandomUUIDGenerator(randomSeed: Long) {
+  private val random = new MersenneTwister(randomSeed)
+
+  def getNextUUID(): UUID = {
--- End diff --

Perhaps we should also create a version that creates a  UTF8String directly.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20817: [SPARK-23599][SQL] Add a UUID generator from Pseu...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20817#discussion_r174665726
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RandomUUIDGenerator.scala
 ---
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import java.util.UUID
+
+import scala.util.Random
+
+/**
+ * This class is used to generate a UUID from Pseudo-Random Numbers 
produced by
+ * Scala Random.
+ *
+ * For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) 
URN Namespace,
+ * section 4.4 "Algorithms for Creating a UUID from Truly Random or 
Pseudo-Random Numbers".
+ */
+case class RandomUUIDGenerator(random: Random) {
+  def getNextUUID(): UUID = {
+val mostSigBits = (random.nextLong() & 0x0FFFL) | 
0x4000L
+val leastSigBits = (random.nextLong() | 0x8000L) & 
0xBFFFL
+
+new UUID(mostSigBits, leastSigBits)
--- End diff --

Ok. Mersenne Twister is used in the update.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20817: [SPARK-23599][SQL] Add a UUID generator from Pseu...

[hvanhovell]

Github user hvanhovell commented on a diff in the pull request:

https://github.com/apache/spark/pull/20817#discussion_r174416499
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RandomUUIDGenerator.scala
 ---
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.util
+
+import java.util.UUID
+
+import scala.util.Random
+
+/**
+ * This class is used to generate a UUID from Pseudo-Random Numbers 
produced by
+ * Scala Random.
+ *
+ * For the algorithm, see RFC 4122: A Universally Unique IDentifier (UUID) 
URN Namespace,
+ * section 4.4 "Algorithms for Creating a UUID from Truly Random or 
Pseudo-Random Numbers".
+ */
+case class RandomUUIDGenerator(random: Random) {
+  def getNextUUID(): UUID = {
+val mostSigBits = (random.nextLong() & 0x0FFFL) | 
0x4000L
+val leastSigBits = (random.nextLong() | 0x8000L) & 
0xBFFFL
+
+new UUID(mostSigBits, leastSigBits)
--- End diff --

I think we need to use a different RNG. `java.util.Random` only has 48 bits 
of state, which is less than the 122 bits we need for random number generation. 
Something like [PCG](http://pcg-random.org/) or a Mersenne twister would work.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20817: [SPARK-23599][SQL] Add a UUID generator from Pseu...

[viirya]

GitHub user viirya opened a pull request:

https://github.com/apache/spark/pull/20817

[SPARK-23599][SQL] Add a UUID generator from Pseudo-Random Numbers

## What changes were proposed in this pull request?

This patch adds a UUID generator from Pseudo-Random Numbers. We can use it 
later to have deterministic `UUID()` expression.

## How was this patch tested?

Added unit tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 SPARK-23599

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20817.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20817


commit b7bce25f83622c4adb97dd03e7414811770d7d4d
Author: Liang-Chi Hsieh 
Date:   2018-03-14T05:31:13Z

Add a UUID generator from Pseudo-Random Numbers.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org