Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21537
I think we can set up a place (mailling list or JIRA) to discuss the
further thing about IR design, as suggested by @HyukjinKwon. This can be a
co-work from interesting parties
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21868#discussion_r210799970
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -459,6 +460,29 @@ object SQLConf {
.intConf
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21537
If we will continue on improving current codegen framework, I think it is
good to have a design doc reviewed by the community.
If we decide to have IR design and get rid of this string based
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21868#discussion_r210785081
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -459,6 +458,29 @@ object SQLConf {
.intConf
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21868#discussion_r210779310
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -459,6 +458,29 @@ object SQLConf {
.intConf
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21868#discussion_r210765335
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -25,17 +25,16 @@ import java.util.zip.Deflater
import
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21537
Thanks for all involving in this discussion!
Sorry that I was on a long flight and feels too tired to reply now. I just
want to say for now, no matter what decision we made, to continue to
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22024
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21537
Thanks @HyukjinKwon
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
Yeah. The encoding rules regarding `Option` are:
1. For non `Product` types, no change is made.
```
Option[Int] in normal encoder -> a nullable int column
Option[Int] in
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21537
@HyukjinKwon Thanks! I've updated. Please let me know if they are good now.
---
-
To unsubscribe, e-mail: reviews-uns
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21439
I think R side is not update for this yet. @huaxingao would you like to do
that?
---
-
To unsubscribe, e-mail: reviews-unsubscr
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21859
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21980
Thanks2! @HyukjinKwon
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209486478
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21439
LGTM too.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21439
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21859
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22076
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209464591
--- Diff: python/pyspark/worker.py ---
@@ -275,6 +280,10 @@ def main(infile, outfile):
shuffle.DiskBytesSpilled = 0
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209464569
--- Diff: python/pyspark/worker.py ---
@@ -261,6 +263,9 @@ def main(infile, outfile):
# initialize global state
taskContext
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209464621
--- Diff: python/pyspark/taskcontext.py ---
@@ -29,6 +29,7 @@ class TaskContext(object):
"""
_tas
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209464679
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -180,7 +183,42 @@ private[spark] abstract class BasePythonRunner[IN
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22085#discussion_r209464515
--- Diff:
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala ---
@@ -180,7 +183,42 @@ private[spark] abstract class BasePythonRunner[IN
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21537#discussion_r209464356
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
---
@@ -1024,26 +1033,29 @@ case class Cast(child: Expression
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22037
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209418749
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209418206
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209417666
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -166,9 +169,17 @@ class RangePartitioner[K : Ordering : ClassTag, V
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209417629
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209410871
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209407875
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -166,9 +169,17 @@ class RangePartitioner[K : Ordering : ClassTag, V
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r209408395
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21980
Thanks @zsxwing
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21977#discussion_r209374683
--- Diff: python/pyspark/worker.py ---
@@ -259,6 +260,26 @@ def main(infile, outfile):
"PYSPARK_DRIVER_PYTHON are corr
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21537
Thanks @mgaido91
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22066
> @viirya is that effort going on? I can help with the work if you want.
Thanks.
@mgaido91 Yeah, I'm still working on it. One of the PRs #21537 is still
waiting fo
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
Thank you! @hvanhovell
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22019
SGTM too.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21732#discussion_r209144900
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
---
@@ -43,20 +43,17 @@ import
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21847
Few minor comments. LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r209116303
--- Diff:
external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala ---
@@ -725,6 +744,205 @@ class AvroSuite extends QueryTest with
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r209115909
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -87,10 +87,36 @@ class AvroSerializer(rootCatalystType
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21847#discussion_r209115631
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala ---
@@ -87,10 +87,36 @@ class AvroSerializer(rootCatalystType
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21732
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22037#discussion_r209067476
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala ---
@@ -479,6 +481,26 @@ object Decimal {
dec
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22037#discussion_r209083275
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala
---
@@ -139,7 +152,22 @@ object SchemaConverters
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21826
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21535
Thanks @HyukjinKwon @hvanhovell @mgaido91 @kiszk
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22018#discussion_r208784598
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
---
@@ -297,7 +297,7 @@ object InMemoryFileIndex
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22018#discussion_r208779110
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala
---
@@ -297,7 +297,7 @@ object InMemoryFileIndex
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21859
This optimization is only for SQL, but other places also use
`RangePartitioner`. What it can affect other places?
---
-
To
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r208775191
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r208775694
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala
---
@@ -294,7 +296,12 @@ object ShuffleExchangeExec
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r208774474
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2799,6 +2799,26 @@ class SQLQuerySuite extends QueryTest with
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r208775584
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -166,7 +169,16 @@ class RangePartitioner[K : Ordering : ClassTag, V
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r208774179
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2799,6 +2799,26 @@ class SQLQuerySuite extends QueryTest with
Github user viirya closed the pull request at:
https://github.com/apache/spark/pull/22046
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21859#discussion_r208770924
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -166,7 +169,16 @@ class RangePartitioner[K : Ordering : ClassTag, V
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22046
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
I set up a test PR for `VersionsSuite` at #22046.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/22046
[JUST_TEST][NOT_MERGE] Test for VersionsSuite
## What changes were proposed in this pull request?
Few test cases in `VersionsSuite` continue to fail in my one PR. But it
isn't reprodu
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
I don't run into this test failure in `VersionsSuite` locally.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apach
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22041
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
Looks like unrelated test failure at `VersionsSuite`...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21520
@HyukjinKwon Thanks for looking into this. It is based on the comment and
discussion here
https://github.com/apache/spark/pull/21193#discussion_r186627099
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22027
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22008#discussion_r208385422
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,45 @@ object EliminateOuterJoin extends Rule
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22008#discussion_r208353576
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
---
@@ -158,8 +158,9 @@ abstract class Optimizer
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/22008#discussion_r208352309
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala
---
@@ -152,3 +153,45 @@ object EliminateOuterJoin extends Rule
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22027
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22019
Empty string should be treated as null for all non string types?
https://github.com/apache/spark/blob/ef57fdd5b0a6f7f0b6343c91c6983d20bc67fb5b/sql/catalyst/src/main/scala/org/apache/spark/sql
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21991
The failed test is `FlatMapGroupsWithStateSuite.flatMapGroupsWithState`. I
saw it fails some times occasionally. I think it should not be related to this
change. @HyukjinKwon @dbtsai
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21980#discussion_r208078032
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
---
@@ -854,6 +854,26 @@ class StreamingQuerySuite extends
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21991#discussion_r208055413
--- Diff: dev/merge_spark_pr.py ---
@@ -142,6 +142,9 @@ def merge_pr(pr_num, target_ref, title, body,
pr_repo_desc):
distinct_authors[0
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21980#discussion_r207971182
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
---
@@ -75,14 +74,11 @@ class IncrementalExecution
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21984#discussion_r207746389
--- Diff:
external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala
---
@@ -100,6 +103,8 @@ class AvroDeserializer(rootAvroType: Schema
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21948
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21948#discussion_r207725721
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriterFactory.java
---
@@ -33,7 +33,10 @@
public interface
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21980
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/16677
@hvanhovell Shall we consider to include this into 2.4?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21988
And put `[BRANCH-2.3]` into the title of the PR #21989 too?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21952
The regression happens at writing. Looks like when benchmarking writing
time, we don't use `df.count`?
---
-
To unsubscri
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21952
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21952
Ah, finally I can reproduce this. It needs to allocate the array feature
with length 16000. I was reducing it to 1600 and it largely relieve the
regression. `com.databricks.spark.avro` is faster
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21980#discussion_r207442765
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala
---
@@ -75,14 +75,15 @@ class IncrementalExecution
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21977#discussion_r207409267
--- Diff:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
---
@@ -333,7 +340,7 @@ private[spark] class Client(
val
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21977#discussion_r207409771
--- Diff: python/pyspark/worker.py ---
@@ -259,6 +260,26 @@ def main(infile, outfile):
"PYSPARK_DRIVER_PYTHON are corr
GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/21980
[SPARK-25010][SQL] Rand/Randn should produce different values for each
execution in streaming query
## What changes were proposed in this pull request?
Like Uuid in SPARK-24896, Rand and
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21952
@dbtsai This is what I see when testing on Spark 2.3. Compared with above
numbers, seems to me there are no such significant difference as same as your
findings.
```scala
801 - 900 of 9977 matches
Mail list logo
