[spark] Git Push Summary
Repository: spark Updated Tags: refs/tags/v2.2.2-rc2 [created] fc28ba3db - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[2/2] spark git commit: Preparing development version 2.2.3-SNAPSHOT
Preparing development version 2.2.3-SNAPSHOT
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/47958270
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/47958270
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/47958270
Branch: refs/heads/branch-2.2
Commit: 47958270f51d08a076e20e299bb2d86fa1927d26
Parents: fc28ba3
Author: Thomas Graves
Authored: Wed Jun 27 13:55:20 2018 +
Committer: Thomas Graves
Committed: Wed Jun 27 13:55:20 2018 +
--
R/pkg/DESCRIPTION | 2 +-
assembly/pom.xml | 2 +-
common/network-common/pom.xml | 2 +-
common/network-shuffle/pom.xml| 2 +-
common/network-yarn/pom.xml | 2 +-
common/sketch/pom.xml | 2 +-
common/tags/pom.xml | 2 +-
common/unsafe/pom.xml | 2 +-
core/pom.xml | 2 +-
docs/_config.yml | 4 ++--
examples/pom.xml | 2 +-
external/docker-integration-tests/pom.xml | 2 +-
external/flume-assembly/pom.xml | 2 +-
external/flume-sink/pom.xml | 2 +-
external/flume/pom.xml| 2 +-
external/kafka-0-10-assembly/pom.xml | 2 +-
external/kafka-0-10-sql/pom.xml | 2 +-
external/kafka-0-10/pom.xml | 2 +-
external/kafka-0-8-assembly/pom.xml | 2 +-
external/kafka-0-8/pom.xml| 2 +-
external/kinesis-asl-assembly/pom.xml | 2 +-
external/kinesis-asl/pom.xml | 2 +-
external/spark-ganglia-lgpl/pom.xml | 2 +-
graphx/pom.xml| 2 +-
launcher/pom.xml | 2 +-
mllib-local/pom.xml | 2 +-
mllib/pom.xml | 2 +-
pom.xml | 2 +-
python/pyspark/version.py | 2 +-
repl/pom.xml | 2 +-
resource-managers/mesos/pom.xml | 2 +-
resource-managers/yarn/pom.xml| 2 +-
sql/catalyst/pom.xml | 2 +-
sql/core/pom.xml | 2 +-
sql/hive-thriftserver/pom.xml | 2 +-
sql/hive/pom.xml | 2 +-
streaming/pom.xml | 2 +-
tools/pom.xml | 2 +-
38 files changed, 39 insertions(+), 39 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/47958270/R/pkg/DESCRIPTION
--
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index 380b3ef..ad72330 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
Package: SparkR
Type: Package
-Version: 2.2.2
+Version: 2.2.3
Title: R Frontend for Apache Spark
Description: Provides an R Frontend for Apache Spark.
Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
http://git-wip-us.apache.org/repos/asf/spark/blob/47958270/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 775d9f8..f9ec6e7 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
org.apache.spark
spark-parent_2.11
-2.2.2
+2.2.3-SNAPSHOT
../pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/47958270/common/network-common/pom.xml
--
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 017f0f1..55d29d5 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
org.apache.spark
spark-parent_2.11
-2.2.2
+2.2.3-SNAPSHOT
../../pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/47958270/common/network-shuffle/pom.xml
--
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index 17d5216..6d84766 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
org.apache.spark
spark-parent_2.11
-2.2.2
+2.2.3-SNAPSHOT
../../pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/47958270/common/network-yarn/pom.xml
--
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 8d03c86..6228be6 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
org.apache.spark
spark-parent_2.11
-2.2.2
+2.2.3-SNAPSHOT
../../pom.xml
[1/2] spark git commit: Preparing Spark release v2.2.2-rc2
Repository: spark
Updated Branches:
refs/heads/branch-2.2 72575d0bb -> 47958270f
Preparing Spark release v2.2.2-rc2
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fc28ba3d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fc28ba3d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fc28ba3d
Branch: refs/heads/branch-2.2
Commit: fc28ba3db7185e84b6dbd02ad8ef8f1d06b9e3c6
Parents: 72575d0
Author: Thomas Graves
Authored: Wed Jun 27 13:55:11 2018 +
Committer: Thomas Graves
Committed: Wed Jun 27 13:55:11 2018 +
--
R/pkg/DESCRIPTION | 2 +-
assembly/pom.xml | 2 +-
common/network-common/pom.xml | 2 +-
common/network-shuffle/pom.xml| 2 +-
common/network-yarn/pom.xml | 2 +-
common/sketch/pom.xml | 2 +-
common/tags/pom.xml | 2 +-
common/unsafe/pom.xml | 2 +-
core/pom.xml | 2 +-
docs/_config.yml | 4 ++--
examples/pom.xml | 2 +-
external/docker-integration-tests/pom.xml | 2 +-
external/flume-assembly/pom.xml | 2 +-
external/flume-sink/pom.xml | 2 +-
external/flume/pom.xml| 2 +-
external/kafka-0-10-assembly/pom.xml | 2 +-
external/kafka-0-10-sql/pom.xml | 2 +-
external/kafka-0-10/pom.xml | 2 +-
external/kafka-0-8-assembly/pom.xml | 2 +-
external/kafka-0-8/pom.xml| 2 +-
external/kinesis-asl-assembly/pom.xml | 2 +-
external/kinesis-asl/pom.xml | 2 +-
external/spark-ganglia-lgpl/pom.xml | 2 +-
graphx/pom.xml| 2 +-
launcher/pom.xml | 2 +-
mllib-local/pom.xml | 2 +-
mllib/pom.xml | 2 +-
pom.xml | 2 +-
python/pyspark/version.py | 2 +-
repl/pom.xml | 2 +-
resource-managers/mesos/pom.xml | 2 +-
resource-managers/yarn/pom.xml| 2 +-
sql/catalyst/pom.xml | 2 +-
sql/core/pom.xml | 2 +-
sql/hive-thriftserver/pom.xml | 2 +-
sql/hive/pom.xml | 2 +-
streaming/pom.xml | 2 +-
tools/pom.xml | 2 +-
38 files changed, 39 insertions(+), 39 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/fc28ba3d/R/pkg/DESCRIPTION
--
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index ad72330..380b3ef 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
Package: SparkR
Type: Package
-Version: 2.2.3
+Version: 2.2.2
Title: R Frontend for Apache Spark
Description: Provides an R Frontend for Apache Spark.
Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),
http://git-wip-us.apache.org/repos/asf/spark/blob/fc28ba3d/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index f9ec6e7..775d9f8 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
org.apache.spark
spark-parent_2.11
-2.2.3-SNAPSHOT
+2.2.2
../pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/fc28ba3d/common/network-common/pom.xml
--
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 55d29d5..017f0f1 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
org.apache.spark
spark-parent_2.11
-2.2.3-SNAPSHOT
+2.2.2
../../pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/fc28ba3d/common/network-shuffle/pom.xml
--
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index 6d84766..17d5216 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
org.apache.spark
spark-parent_2.11
-2.2.3-SNAPSHOT
+2.2.2
../../pom.xml
http://git-wip-us.apache.org/repos/asf/spark/blob/fc28ba3d/common/network-yarn/pom.xml
--
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 6228be6..8d03c86 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
org.apache.spark
spark-parent_2.11
-2.2.3-SNAPSHOT
+2.2.2
../../pom.xml
svn commit: r27768 - /dev/spark/v2.2.2-rc2-bin/
Author: tgraves Date: Wed Jun 27 17:16:26 2018 New Revision: 27768 Log: Apache Spark v2.2.2-rc2 Added: dev/spark/v2.2.2-rc2-bin/ dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz (with props) dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.asc dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.sha512 dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz (with props) dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.asc dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.sha512 dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-hadoop2.6.tgz (with props) dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-hadoop2.6.tgz.asc dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-hadoop2.6.tgz.sha512 dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-hadoop2.7.tgz (with props) dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-hadoop2.7.tgz.asc dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-hadoop2.7.tgz.sha512 dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-without-hadoop.tgz (with props) dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-without-hadoop.tgz.asc dev/spark/v2.2.2-rc2-bin/spark-2.2.2-bin-without-hadoop.tgz.sha512 dev/spark/v2.2.2-rc2-bin/spark-2.2.2.tgz (with props) dev/spark/v2.2.2-rc2-bin/spark-2.2.2.tgz.asc dev/spark/v2.2.2-rc2-bin/spark-2.2.2.tgz.sha512 Added: dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz == Binary file - no diff available. Propchange: dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz -- svn:mime-type = application/octet-stream Added: dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.asc == --- dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.asc (added) +++ dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.asc Wed Jun 27 17:16:26 2018 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1 + +iQIcBAABAgAGBQJbM576AAoJEN5PvM2B5sdqBcQP/390/0Cyo/E2UYUs361AkaR2 +jCI+w0/+msaX4wd0TExllr2BG2Cwz2XHxL0nIOtd3h8MKAw/iBwO2eVd2imHa27i +iZkFXrEBT8Gsompq6Nl/j4Cht/I5ay5qEaG4QHSTnJbLgtQIVVw5Rovw/KLVVm0/ +9r+YYFjahsFTw3JTp62LFUXbF81aTgJWwSw05b5TvAwfws4TZoZmES8yz/YKrRWe +UBZFShAvehVEkSqP7qYm8Dd/fQnfyxaTTUfWXcJQX7fHNrQGXdPOECnlC+TMrmn2 +AqQaKTjHU3p8Wvv78tVLmal9By0NWcPqiSrQ2TlJFEygKseqrDNTw47p06xZHuuz +bYmJu5k7JcQJyG7ThpQgnIFga0TjM4Ymm+9pN5OKsPyNIGhFmWHVp/zAkdvMHVUg +D+5KIDJZPtOK3iL8hAjvgCM7W+Q0GcSOCtigNc7S3CaP+0xcI7RfOAy/Ns4wHPwD +hJZA6u8lLntVwYfmamW4SCB6WHDvdHXl8H2QQ+mVqRpGdnK2fYBUpvcs0dijUAs9 +ZWIvDPuvCE+SnW3zShY0CGxwAhYFEO6vXVExQWsdInTlG7qXDtJ2S+9shm8PohYy +CdDijjBW5sVR9OnLEFoMtxiOGXaGNo04kK4U8lzkq5reFtQc8I9fr+TUSvki9WqM +Mf1RUFRr959InxcvafD8 +=WMWJ +-END PGP SIGNATURE- Added: dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.sha512 == --- dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.sha512 (added) +++ dev/spark/v2.2.2-rc2-bin/SparkR_2.2.2.tar.gz.sha512 Wed Jun 27 17:16:26 2018 @@ -0,0 +1,3 @@ +SparkR_2.2.2.tar.gz: FD8AB48E 7379B62C 85744DD8 D9CC0C85 C76EA843 460C3FEC + A65DBA30 A47B427C A388A8E2 99CF593A 0CE70FA6 F66BA6BF + 2280DF30 F3C4FA63 E451C4BD D49D124D Added: dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz == Binary file - no diff available. Propchange: dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz -- svn:mime-type = application/octet-stream Added: dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.asc == --- dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.asc (added) +++ dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.asc Wed Jun 27 17:16:26 2018 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v1 + +iQIcBAABAgAGBQJbM6QPAAoJEN5PvM2B5sdqnhsP/0pwzvbnrycc6EgAr0/Qnzeu +IDc56d+NIOFLX3KcqF+ggUpll5tj6+Llq8Qs+JNz4ufBT5FNd6/0sot9PfU86AzN +A5+mlLAyOUTh1WSxY54Mo3Mc8nrLKQHw7QQFMxCfay9oTBrlURhyNp5bEwDfOurf +LBlX36sTv3H60N9/RZQWt2OF9YnGHntLIe2Fsqk7GtnUMmkdAqdGGtq1w4cjVtci +5/8xRqRwhVoaGcLLWzsdYMuOqnhBXq6GKtCgF4PLy25vAPVpVQmwcwrx2GVSY6Rr +M5RNGw8cq2AdPHGojbygUtwg7h2N8BFqwM5EgYKIaKyWtw0N7RKRJ4YWl7QJorAe +Q4xfFGCwmzmwqMpl3/ZontQMw56FedWLVuETmBoE30/lqPZKsJCU9eS0/8pdBo4t +7UpQgao25kaBqy4RS2A48343oY9y4vreUQxxjxilV/RxCYI+yNalU3+oreHpaDAk +k9SGprdmt5zXAkEd4wNmrLWHVB/IlyzjH8s7gKBlHA9GLpxnFwlMInZJjZkN7Jxt +6A3SDOzhuuEkPHjb5rT1DyCFv7w5rN6TZ0iFxAh9sgR5MbMBaQFRr8krPiYVcG/2 +k9d8FMrDgAfttRVMe6fOg721bnAZaDwdCcV07DMn5/lR60bjhhHMrsD7JWy89RI9 +qrS3ckY1stERB3toI/Li +=yKed +-END PGP SIGNATURE- Added: dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.sha512 == --- dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.sha512 (added) +++ dev/spark/v2.2.2-rc2-bin/pyspark-2.2.2.tar.gz.sha512
spark git commit: [SPARK-24215][PYSPARK][FOLLOW UP] Implement eager evaluation for DataFrame APIs in PySpark
Repository: spark
Updated Branches:
refs/heads/master a1a64e358 -> 6a0b77a55
[SPARK-24215][PYSPARK][FOLLOW UP] Implement eager evaluation for DataFrame APIs
in PySpark
## What changes were proposed in this pull request?
Address comments in #21370 and add more test.
## How was this patch tested?
Enhance test in pyspark/sql/test.py and DataFrameSuite
Author: Yuanjian Li
Closes #21553 from xuanyuanking/SPARK-24215-follow.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6a0b77a5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6a0b77a5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6a0b77a5
Branch: refs/heads/master
Commit: 6a0b77a55d53e74ac0a0892556c3a7a933474948
Parents: a1a64e3
Author: Yuanjian Li
Authored: Wed Jun 27 10:43:06 2018 -0700
Committer: Xiao Li
Committed: Wed Jun 27 10:43:06 2018 -0700
--
docs/configuration.md | 27 -
python/pyspark/sql/dataframe.py | 3 +-
python/pyspark/sql/tests.py | 46 ++-
.../org/apache/spark/sql/internal/SQLConf.scala | 23
.../scala/org/apache/spark/sql/Dataset.scala| 11 ++--
.../org/apache/spark/sql/DataFrameSuite.scala | 59
6 files changed, 131 insertions(+), 38 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6a0b77a5/docs/configuration.md
--
diff --git a/docs/configuration.md b/docs/configuration.md
index 6aa7878..0c7c447 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -457,33 +457,6 @@ Apart from these, the following properties are also
available, and may be useful
- spark.sql.repl.eagerEval.enabled
- false
-
-Enable eager evaluation or not. If true and the REPL you are using
supports eager evaluation,
-Dataset will be ran automatically. The HTML table which generated by
_repl_html_
-called by notebooks like Jupyter will feedback the queries user have
defined. For plain Python
-REPL, the output will be shown like dataframe.show()
-(see https://issues.apache.org/jira/browse/SPARK-24215;>SPARK-24215 for
more details).
-
-
-
- spark.sql.repl.eagerEval.maxNumRows
- 20
-
-Default number of rows in eager evaluation output HTML table generated by
_repr_html_ or plain text,
-this only take effect when spark.sql.repl.eagerEval.enabled
is set to true.
-
-
-
- spark.sql.repl.eagerEval.truncate
- 20
-
-Default number of truncate in eager evaluation output HTML table generated
by _repr_html_ or
-plain text, this only take effect when
spark.sql.repl.eagerEval.enabled set to true.
-
-
-
spark.files
http://git-wip-us.apache.org/repos/asf/spark/blob/6a0b77a5/python/pyspark/sql/dataframe.py
--
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py
index 1e6a1ac..cb3fe44 100644
--- a/python/pyspark/sql/dataframe.py
+++ b/python/pyspark/sql/dataframe.py
@@ -393,9 +393,8 @@ class DataFrame(object):
self._support_repr_html = True
if self._eager_eval:
max_num_rows = max(self._max_num_rows, 0)
-vertical = False
sock_info = self._jdf.getRowsToPython(
-max_num_rows, self._truncate, vertical)
+max_num_rows, self._truncate)
rows = list(_load_from_socket(sock_info,
BatchedSerializer(PickleSerializer(
head = rows[0]
row_data = rows[1:]
http://git-wip-us.apache.org/repos/asf/spark/blob/6a0b77a5/python/pyspark/sql/tests.py
--
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py
index 35a0636..8d73806 100644
--- a/python/pyspark/sql/tests.py
+++ b/python/pyspark/sql/tests.py
@@ -3351,11 +3351,41 @@ class SQLTests(ReusedSQLTestCase):
finally:
shutil.rmtree(path)
-def test_repr_html(self):
+def test_repr_behaviors(self):
import re
pattern = re.compile(r'^ *\|', re.MULTILINE)
df = self.spark.createDataFrame([(1, "1"), (2, "2")], ("key",
"value"))
-self.assertEquals(None, df._repr_html_())
+
+# test when eager evaluation is enabled and _repr_html_ will not be
called
+with self.sql_conf({"spark.sql.repl.eagerEval.enabled": True}):
+expected1 = """+-+-+
+|| key|value|
+|+-+-+
+||1|1|
+||2|2|
+|+-+-+
+|"""
+self.assertEquals(re.sub(pattern, '', expected1), df.__repr__())
+with
svn commit: r27772 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_27_12_01-c04cb2d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Wed Jun 27 19:15:40 2018 New Revision: 27772 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_27_12_01-c04cb2d docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r27769 - in /dev/spark/v2.2.2-rc2-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _site/api/java/org/apache/spark
Author: tgraves Date: Wed Jun 27 17:44:22 2018 New Revision: 27769 Log: Apache Spark v2.2.2-rc2 docs [This commit notification would consist of 1346 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
Repository: spark
Updated Branches:
refs/heads/master 78ecb6d45 -> c04cb2d1b
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
## What changes were proposed in this pull request?
Set createTime for every hive partition created in Spark SQL, which could be
used to manage data lifecycle in Hive warehouse. We found that almost every
partition modified by spark sql has not been set createTime.
```
mysql> select * from partitions where create_time=0 limit 1\G;
*** 1. row ***
PART_ID: 1028584
CREATE_TIME: 0
LAST_ACCESS_TIME: 1502203611
PART_NAME: date=20170130
SD_ID: 1543605
TBL_ID: 211605
LINK_TARGET_ID: NULL
1 row in set (0.27 sec)
```
## How was this patch tested?
N/A
Author: debugger87
Author: Chaozhong Yang
Closes #18900 from debugger87/fix/set-create-time-for-hive-partition.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c04cb2d1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c04cb2d1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c04cb2d1
Branch: refs/heads/master
Commit: c04cb2d1b72b1edaddf684755f5a9d6aaf00e03b
Parents: 78ecb6d
Author: debugger87
Authored: Wed Jun 27 11:34:28 2018 -0700
Committer: Xiao Li
Committed: Wed Jun 27 11:34:28 2018 -0700
--
.../apache/spark/sql/catalyst/catalog/interface.scala | 6 ++
.../sql/catalyst/catalog/SessionCatalogSuite.scala| 6 --
.../results/describe-part-after-analyze.sql.out | 14 ++
.../test/resources/sql-tests/results/describe.sql.out | 4
.../resources/sql-tests/results/show-tables.sql.out | 2 ++
.../apache/spark/sql/hive/client/HiveClientImpl.scala | 4
6 files changed, 34 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c04cb2d1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
index f3e67dc..c6105c5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
@@ -93,12 +93,16 @@ object CatalogStorageFormat {
* @param spec partition spec values indexed by column name
* @param storage storage format of the partition
* @param parameters some parameters for the partition
+ * @param createTime creation time of the partition, in milliseconds
+ * @param lastAccessTime last access time, in milliseconds
* @param stats optional statistics (number of rows, total size, etc.)
*/
case class CatalogTablePartition(
spec: CatalogTypes.TablePartitionSpec,
storage: CatalogStorageFormat,
parameters: Map[String, String] = Map.empty,
+createTime: Long = System.currentTimeMillis,
+lastAccessTime: Long = -1,
stats: Option[CatalogStatistics] = None) {
def toLinkedHashMap: mutable.LinkedHashMap[String, String] = {
@@ -109,6 +113,8 @@ case class CatalogTablePartition(
if (parameters.nonEmpty) {
map.put("Partition Parameters", s"{${parameters.map(p => p._1 + "=" +
p._2).mkString(", ")}}")
}
+map.put("Created Time", new Date(createTime).toString)
+map.put("Last Access", new Date(lastAccessTime).toString)
stats.foreach(s => map.put("Partition Statistics", s.simpleString))
map
}
http://git-wip-us.apache.org/repos/asf/spark/blob/c04cb2d1/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
--
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
index 6abab00..6a7375e 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
@@ -1114,11 +1114,13 @@ abstract class SessionCatalogSuite extends AnalysisTest
{
// And for hive serde table, hive metastore will set some
values(e.g.transient_lastDdlTime)
// in table's parameters and storage's properties, here we also ignore
them.
val actualPartsNormalize = actualParts.map(p =>
- p.copy(parameters = Map.empty, storage = p.storage.copy(
+ p.copy(parameters = Map.empty, createTime = -1, lastAccessTime = -1,
+storage = p.storage.copy(
properties = Map.empty, locationUri = None, serde =
svn commit: r27773 - /release/spark/KEYS
Author: tgraves Date: Wed Jun 27 20:04:43 2018 New Revision: 27773 Log: Update KEYS Modified: release/spark/KEYS Modified: release/spark/KEYS == --- release/spark/KEYS (original) +++ release/spark/KEYS Wed Jun 27 20:04:43 2018 @@ -589,3 +589,60 @@ pp4VlK8tOgCZ3o6bWPR95GvVBvALnpoOJjS+cGwO 7fva6arUhIPxFUb+7G0SqJkpib8s4IZ+os3wh4Rv5ZfDAavB4ftC298= =DFmT -END PGP PUBLIC KEY BLOCK- + +pub 4096R/81E6C76A 2018-06-13 +uid Thomas Graves (CODE SIGNING KEY) +sub 4096R/FC8689C4 2018-06-13 + +-BEGIN PGP PUBLIC KEY BLOCK- +Version: GnuPG v2.0.22 (GNU/Linux) + +mQINBFsgdxABEACwfMYBgIW9v73TIIQJnpHWE7SgGHfAs60FR128uLzzcAJcAly6 +/lq4nRU6inNqiATM3f/HXKhKsv1koFtqPXz3yT6SYTgbMLP0zCU41nPi00r0AOY0 +KrRmM76bbx1hYVjw3L0gTz2froeQxcYoTUMPNqGFjGfEYHfWo04O2Ev5nf+P5kEU +btk+QhzetcmIKRHVg1DSivPSknGYFRjVREWJTsfz8dZ/2MkN3gqPDcrIIgHJrGpM +cAESFz1E7RLX3i+cJGH90BCDXv2DgcRtVAHJ/9b9IbfZ24pWqwk7nt7tpcGMwDPy +9caQ2H5q4H+ECd2Rj2Tzh+ptRz6csLWUAONlS/+rP208wVTNqNCONlw5euYh8wJe +DVrahHUlsI/SB5edBKQY6/nW+54wphHdbF1iE7GV0XFyG4t0RARkvJlaO+5SMngw +3nxMd/CoL8r0j+0oncUkb8ZAPzJaVs0YHtEKOf/k7PpaRZW5LYCxiZzZWcjyTxtT +kqrPZ11MwEqLVFiEMnEg4JVCX14WqP9AdWnrw1pOf9iAgs7LwyOEWxaR9CN70CLK +Qao56vILC5/B9FN4jyee5y5U8yujqIXjU1ptKnMmYDEZn/fvns0kOix//lWSdHg5 +yt6/95Q/QXxOo4tenlKvLfORVkY4RNmWUOErZ+hbu0cysS8YToOif4+F1QARAQAB +tDVUaG9tYXMgR3JhdmVzIChDT0RFIFNJR05JTkcgS0VZKSA8dGdyYXZlc0BhcGFj +aGUub3JnPokCNwQTAQoAIQUCWyB3EAIbAwULCQgHAwUVCgkICwUWAgMBAAIeAQIX +gAAKCRDeT7zNgebHam5pEACHgEA+xEsL9TYKj7lk5pwa7tVSFRIjiZqu7fausssS +sAnCDyHv683HKLh8b+tFONeZlTIz1ygeEnbzYr2uthF8GPId1ydNFrZ3/tV/8XlN +hb2AwThuDXvMHpw1GvTqbMF80kdJ5SIUIC8l6M05NyInU9kTF3mKjol2UXIpBono +rcSeozCn6k6NLYAopmwJMGtMc4vf1SrCn59+Y2QUBrUsw8iA1yAwc2IYdteYbC1k +4lrLPIMvGzKC8wAb2I46B3EWkR4soQ8VYfdkZKjJqdcYUSJSL9Jw2XgRuQxKrSoB +6tc52eR0V/rPCJCTkqFB5eLIYwn4A1oD3PZllSaBlG6UxWlxZiyoAf6EzgIbORTg +ark5NA6r9p+is7U684ysLBa1be/RX4nq7OIxdOomARErVbepDsJa8TkTyGs+CTau +AWIi9UbYn9DZ08VMe0PPqT+7s2kcjuddIP2WX6ScYt0c3HAAr9Ky3FwOGvkMlWo2 +DETmQwt1RNhEZC0kSvJSveqHmMaaY4DcgcfPYqUpYbLKDiIzxoKkAOYeyt7X85qJ +xpUC5nbA1G9Vp1o+zaqzOqAOLTLsI370kVj+57XoFiwIo9Fl0VlSxLoCzXJ7Rjkt +iCu2Wwxd22W4f8OtRVXJQsL1ikAa5hsE0zkpHHypa8Wa41NqVM2H1nPjOLmcRQky +xLkCDQRbIHcQARAA5C1Z49Fs0PsKQNjEFyMxvJGCAdGhQmH4VGqZxJQ5GRmScucG +gPkeW2L3n3v6oguASihB8KgF+WgLgM/8Tn5u7BacY+7hcqgSDgUglVAqcGGrsNTh +PTdO92HFcIdKfAoacF8IPRDwhCBABVUQyfhVS4yzTHbk1JbX80mcGOkaZMU1tnrF +etfxcxEQI+1DP/JZ70A6EK+FlyGgmtjrjPRV38DJWDYV4NxHEXSO0d669dIroHBE +LzcbarB4k9tkSnO2hmbulX2622vKKMuxDmIbKgid69AKVWD0qFaCNSYrdOcexbQr ++LNW6gS30sHhauUIvAu1pGWJefwJjvSme39uBbhG9dpP9tjeUH62RejXvqu0lb4o +t8tRbWcOZXEZhQQy/SVJXwpMevGAKO8euOuIY9m3uNzf0wMKD7y/023odDhk4qb7 +OIjJwq1NJqkIRJLaY5MBeyAd7xwkfi0/EhrSkaoISJwbrrfHaBCgEJVBWXF+pRCB +Tp6FotqL45AH/4j+/cw/k94kqc433u7P1ORCWWwgVxQN69X+8hZBo5BSVnW+TSvY +MATFM/7uMvj2yk0DHPbK5jZ2sqpe5ejQTmougya3mj8kEr/vyNExM8ebo+KdSEuZ +hnbBjRhCsQ/SgwIpY53Lbp4gZAPjN66VX0kbuJ0Nocvp2dMl+9QkVq+9im8AEQEA +AYkCHwQYAQoACQUCWyB3EAIbDAAKCRDeT7zNgebHavfsEACp7nEkv7fYU9QgnGpP +/14yICPtnfBJzJ5Z35WkjSStBxd+Xdbnec1UlLqzGPPuXtLOcouRvmN6WXGPynB9 +ERd2Pga1yiexDZhVVR/vucwNq5pQv6LTlufMI1Nan/p6swwlsDWpVzPfGRaOMOt1 +1OaWeTA0q2ByDWQQV1qay09fs9dJsW6YBzFCKwNNaoXhM5zrmlQj80Ces7gMBJd+ +iivRVceRC+b2vAMvceX2FReS21o4S+mTKlTCvUNDbJe46a+rUEeHzBYN5qvbLedO +PQEXXUJLrjbj8/uA5gfW+Ib2azfpo7ezPnxYBNz4UC9CGF3JWFnYX7UHpwniB2pt +J9RrNiKL9Ohdpg1qNgJCAXzbrE31bqpaGawTFYMtRlEYHKDTs6jglX9cIVqDiz+y +smiMjVBM4G8PVwCvC0OfuCOqQrbieiSNfdBlcWJG5MbZqyXwQNiuRl5tdJPtB4Kx +NfSm3khyy1/xze0gOi4OwjznHXwW7606Uu6eAKKhWcQ7mHovEJSp5ziWb+/1PXe7 +f5XpIqrYxuMpghlA6Q72sON9zcmFEKW6y+3mRKtBzOsJXLKS0AWXi19zoFlq3Kzs +UiZuJIMHLPJK4sbOj5nMZE2163zUXz+gOVMeLqVxV/zlW6xnYu84taF4PbYIzmts +qG1EJoF+tteqemi1ZwYipD06wA== +=cA5d +-END PGP PUBLIC KEY BLOCK- - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24446][YARN] Properly quote library path for YARN.
Repository: spark
Updated Branches:
refs/heads/master 6a0b77a55 -> 78ecb6d45
[SPARK-24446][YARN] Properly quote library path for YARN.
Because the way YARN executes commands via bash -c, everything needs
to be quoted so that the whole command is fully contained inside a
bash string and is interpreted correctly when the string is read by
bash. This is a bit different than the quoting done when executing
things as if typing in a bash shell.
Tweaked unit tests to exercise the bad behavior, which would cause
existing tests to time out without the fix. Also tested on a real
cluster, verifying the shell script created by YARN to run the
container.
Author: Marcelo Vanzin
Closes #21476 from vanzin/SPARK-24446.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/78ecb6d4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/78ecb6d4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/78ecb6d4
Branch: refs/heads/master
Commit: 78ecb6d457970b136a2e0e0e27d170c84ea28eac
Parents: 6a0b77a
Author: Marcelo Vanzin
Authored: Wed Jun 27 10:57:29 2018 -0700
Committer: Marcelo Vanzin
Committed: Wed Jun 27 10:57:29 2018 -0700
--
.../org/apache/spark/deploy/yarn/Client.scala | 22 ++--
.../spark/deploy/yarn/ExecutorRunnable.scala| 11 +-
.../deploy/yarn/BaseYarnClusterSuite.scala | 9
3 files changed, 34 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/78ecb6d4/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
--
diff --git
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
index 7225ff0..793d012 100644
---
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
+++
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
@@ -899,7 +899,8 @@ private[spark] class Client(
val libraryPaths = Seq(sparkConf.get(DRIVER_LIBRARY_PATH),
sys.props.get("spark.driver.libraryPath")).flatten
if (libraryPaths.nonEmpty) {
-prefixEnv = Some(getClusterPath(sparkConf,
Utils.libraryPathEnvPrefix(libraryPaths)))
+prefixEnv =
Some(createLibraryPathPrefix(libraryPaths.mkString(File.pathSeparator),
+ sparkConf))
}
if (sparkConf.get(AM_JAVA_OPTIONS).isDefined) {
logWarning(s"${AM_JAVA_OPTIONS.key} will not take effect in cluster
mode")
@@ -921,7 +922,7 @@ private[spark] class Client(
.map(YarnSparkHadoopUtil.escapeForShell)
}
sparkConf.get(AM_LIBRARY_PATH).foreach { paths =>
-prefixEnv = Some(getClusterPath(sparkConf,
Utils.libraryPathEnvPrefix(Seq(paths
+prefixEnv = Some(createLibraryPathPrefix(paths, sparkConf))
}
}
@@ -1485,6 +1486,23 @@ private object Client extends Logging {
YarnAppReport(report.getYarnApplicationState(),
report.getFinalApplicationStatus(), diagsOpt)
}
+ /**
+ * Create a properly quoted and escaped library path string to be added as a
prefix to the command
+ * executed by YARN. This is different from normal quoting / escaping due to
YARN executing the
+ * command through "bash -c".
+ */
+ def createLibraryPathPrefix(libpath: String, conf: SparkConf): String = {
+val cmdPrefix = if (Utils.isWindows) {
+ Utils.libraryPathEnvPrefix(Seq(libpath))
+} else {
+ val envName = Utils.libraryPathEnvName
+ // For quotes, escape both the quote and the escape character when
encoding in the command
+ // string.
+ val quoted = libpath.replace("\"", "\\\"")
+ envName + "=\\\"" + quoted + File.pathSeparator + "$" + envName + "\\\""
+}
+getClusterPath(conf, cmdPrefix)
+ }
}
private[spark] class YarnClusterApplication extends SparkApplication {
http://git-wip-us.apache.org/repos/asf/spark/blob/78ecb6d4/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
--
diff --git
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
index a2a18cd..49a0b93 100644
---
a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
+++
b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
@@ -131,10 +131,6 @@ private[yarn] class ExecutorRunnable(
// Extra options for the JVM
val javaOpts = ListBuffer[String]()
-// Set the environment variable through a command prefix
-// to
svn commit: r27778 - in /dev/spark/2.3.2-SNAPSHOT-2018_06_27_14_01-6e1f5e0-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Wed Jun 27 21:15:23 2018 New Revision: 27778 Log: Apache Spark 2.3.2-SNAPSHOT-2018_06_27_14_01-6e1f5e0 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24613][SQL] Cache with UDF could not be matched with subsequent dependent caches
Repository: spark
Updated Branches:
refs/heads/branch-2.3 db538b25a -> 6e1f5e018
[SPARK-24613][SQL] Cache with UDF could not be matched with subsequent
dependent caches
Wrap the logical plan with a `AnalysisBarrier` for execution plan compilation
in CacheManager, in order to avoid the plan being analyzed again.
Add one test in `DatasetCacheSuite`
Author: Maryann Xue
Closes #21602 from maryannxue/cache-mismatch.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6e1f5e01
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6e1f5e01
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6e1f5e01
Branch: refs/heads/branch-2.3
Commit: 6e1f5e0182e1d111f7252c24c576674d1d2c7b91
Parents: db538b2
Author: Maryann Xue
Authored: Thu Jun 21 11:45:30 2018 -0700
Committer: Xiao Li
Committed: Wed Jun 27 13:20:57 2018 -0700
--
.../apache/spark/sql/execution/CacheManager.scala | 6 +++---
.../org/apache/spark/sql/DatasetCacheSuite.scala| 16
2 files changed, 19 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6e1f5e01/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
index d68aeb2..dbdda27 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala
@@ -26,7 +26,7 @@ import org.apache.hadoop.fs.{FileSystem, Path}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.{Dataset, SparkSession}
import org.apache.spark.sql.catalyst.expressions.SubqueryExpression
-import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, ResolvedHint}
+import org.apache.spark.sql.catalyst.plans.logical.{AnalysisBarrier,
LogicalPlan, ResolvedHint}
import org.apache.spark.sql.execution.columnar.InMemoryRelation
import org.apache.spark.sql.execution.datasources.{HadoopFsRelation,
LogicalRelation}
import org.apache.spark.storage.StorageLevel
@@ -97,7 +97,7 @@ class CacheManager extends Logging {
val inMemoryRelation = InMemoryRelation(
sparkSession.sessionState.conf.useCompression,
sparkSession.sessionState.conf.columnBatchSize, storageLevel,
-sparkSession.sessionState.executePlan(planToCache).executedPlan,
+
sparkSession.sessionState.executePlan(AnalysisBarrier(planToCache)).executedPlan,
tableName,
planToCache.stats)
cachedData.add(CachedData(planToCache, inMemoryRelation))
@@ -146,7 +146,7 @@ class CacheManager extends Logging {
useCompression = cd.cachedRepresentation.useCompression,
batchSize = cd.cachedRepresentation.batchSize,
storageLevel = cd.cachedRepresentation.storageLevel,
- child = spark.sessionState.executePlan(cd.plan).executedPlan,
+ child =
spark.sessionState.executePlan(AnalysisBarrier(cd.plan)).executedPlan,
tableName = cd.cachedRepresentation.tableName,
statsOfPlanToCache = cd.plan.stats)
needToRecache += cd.copy(cachedRepresentation = newCache)
http://git-wip-us.apache.org/repos/asf/spark/blob/6e1f5e01/sql/core/src/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala
index e0561ee..f6c760e 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DatasetCacheSuite.scala
@@ -17,6 +17,7 @@
package org.apache.spark.sql
+import org.apache.spark.sql.execution.columnar.{InMemoryRelation,
InMemoryTableScanExec}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.test.SharedSQLContext
import org.apache.spark.storage.StorageLevel
@@ -96,4 +97,19 @@ class DatasetCacheSuite extends QueryTest with
SharedSQLContext {
agged.unpersist()
assert(agged.storageLevel == StorageLevel.NONE, "The Dataset agged should
not be cached.")
}
+
+ test("SPARK-24613 Cache with UDF could not be matched with subsequent
dependent caches") {
+val udf1 = udf({x: Int => x + 1})
+val df = spark.range(0, 10).toDF("a").withColumn("b", udf1($"a"))
+val df2 = df.agg(sum(df("b")))
+
+df.cache()
+df.count()
+df2.cache()
+
+val plan = df2.queryExecution.withCachedData
+assert(plan.isInstanceOf[InMemoryRelation])
+val internalPlan = plan.asInstanceOf[InMemoryRelation].child
+
svn commit: r27774 - /dev/spark/KEYS
Author: tgraves Date: Wed Jun 27 20:22:34 2018 New Revision: 27774 Log: Update KEYS Modified: dev/spark/KEYS Modified: dev/spark/KEYS == --- dev/spark/KEYS (original) +++ dev/spark/KEYS Wed Jun 27 20:22:34 2018 @@ -589,3 +589,60 @@ pp4VlK8tOgCZ3o6bWPR95GvVBvALnpoOJjS+cGwO 7fva6arUhIPxFUb+7G0SqJkpib8s4IZ+os3wh4Rv5ZfDAavB4ftC298= =DFmT -END PGP PUBLIC KEY BLOCK- + +pub 4096R/81E6C76A 2018-06-13 +uid Thomas Graves (CODE SIGNING KEY) +sub 4096R/FC8689C4 2018-06-13 + +-BEGIN PGP PUBLIC KEY BLOCK- +Version: GnuPG v2.0.22 (GNU/Linux) + +mQINBFsgdxABEACwfMYBgIW9v73TIIQJnpHWE7SgGHfAs60FR128uLzzcAJcAly6 +/lq4nRU6inNqiATM3f/HXKhKsv1koFtqPXz3yT6SYTgbMLP0zCU41nPi00r0AOY0 +KrRmM76bbx1hYVjw3L0gTz2froeQxcYoTUMPNqGFjGfEYHfWo04O2Ev5nf+P5kEU +btk+QhzetcmIKRHVg1DSivPSknGYFRjVREWJTsfz8dZ/2MkN3gqPDcrIIgHJrGpM +cAESFz1E7RLX3i+cJGH90BCDXv2DgcRtVAHJ/9b9IbfZ24pWqwk7nt7tpcGMwDPy +9caQ2H5q4H+ECd2Rj2Tzh+ptRz6csLWUAONlS/+rP208wVTNqNCONlw5euYh8wJe +DVrahHUlsI/SB5edBKQY6/nW+54wphHdbF1iE7GV0XFyG4t0RARkvJlaO+5SMngw +3nxMd/CoL8r0j+0oncUkb8ZAPzJaVs0YHtEKOf/k7PpaRZW5LYCxiZzZWcjyTxtT +kqrPZ11MwEqLVFiEMnEg4JVCX14WqP9AdWnrw1pOf9iAgs7LwyOEWxaR9CN70CLK +Qao56vILC5/B9FN4jyee5y5U8yujqIXjU1ptKnMmYDEZn/fvns0kOix//lWSdHg5 +yt6/95Q/QXxOo4tenlKvLfORVkY4RNmWUOErZ+hbu0cysS8YToOif4+F1QARAQAB +tDVUaG9tYXMgR3JhdmVzIChDT0RFIFNJR05JTkcgS0VZKSA8dGdyYXZlc0BhcGFj +aGUub3JnPokCNwQTAQoAIQUCWyB3EAIbAwULCQgHAwUVCgkICwUWAgMBAAIeAQIX +gAAKCRDeT7zNgebHam5pEACHgEA+xEsL9TYKj7lk5pwa7tVSFRIjiZqu7fausssS +sAnCDyHv683HKLh8b+tFONeZlTIz1ygeEnbzYr2uthF8GPId1ydNFrZ3/tV/8XlN +hb2AwThuDXvMHpw1GvTqbMF80kdJ5SIUIC8l6M05NyInU9kTF3mKjol2UXIpBono +rcSeozCn6k6NLYAopmwJMGtMc4vf1SrCn59+Y2QUBrUsw8iA1yAwc2IYdteYbC1k +4lrLPIMvGzKC8wAb2I46B3EWkR4soQ8VYfdkZKjJqdcYUSJSL9Jw2XgRuQxKrSoB +6tc52eR0V/rPCJCTkqFB5eLIYwn4A1oD3PZllSaBlG6UxWlxZiyoAf6EzgIbORTg +ark5NA6r9p+is7U684ysLBa1be/RX4nq7OIxdOomARErVbepDsJa8TkTyGs+CTau +AWIi9UbYn9DZ08VMe0PPqT+7s2kcjuddIP2WX6ScYt0c3HAAr9Ky3FwOGvkMlWo2 +DETmQwt1RNhEZC0kSvJSveqHmMaaY4DcgcfPYqUpYbLKDiIzxoKkAOYeyt7X85qJ +xpUC5nbA1G9Vp1o+zaqzOqAOLTLsI370kVj+57XoFiwIo9Fl0VlSxLoCzXJ7Rjkt +iCu2Wwxd22W4f8OtRVXJQsL1ikAa5hsE0zkpHHypa8Wa41NqVM2H1nPjOLmcRQky +xLkCDQRbIHcQARAA5C1Z49Fs0PsKQNjEFyMxvJGCAdGhQmH4VGqZxJQ5GRmScucG +gPkeW2L3n3v6oguASihB8KgF+WgLgM/8Tn5u7BacY+7hcqgSDgUglVAqcGGrsNTh +PTdO92HFcIdKfAoacF8IPRDwhCBABVUQyfhVS4yzTHbk1JbX80mcGOkaZMU1tnrF +etfxcxEQI+1DP/JZ70A6EK+FlyGgmtjrjPRV38DJWDYV4NxHEXSO0d669dIroHBE +LzcbarB4k9tkSnO2hmbulX2622vKKMuxDmIbKgid69AKVWD0qFaCNSYrdOcexbQr ++LNW6gS30sHhauUIvAu1pGWJefwJjvSme39uBbhG9dpP9tjeUH62RejXvqu0lb4o +t8tRbWcOZXEZhQQy/SVJXwpMevGAKO8euOuIY9m3uNzf0wMKD7y/023odDhk4qb7 +OIjJwq1NJqkIRJLaY5MBeyAd7xwkfi0/EhrSkaoISJwbrrfHaBCgEJVBWXF+pRCB +Tp6FotqL45AH/4j+/cw/k94kqc433u7P1ORCWWwgVxQN69X+8hZBo5BSVnW+TSvY +MATFM/7uMvj2yk0DHPbK5jZ2sqpe5ejQTmougya3mj8kEr/vyNExM8ebo+KdSEuZ +hnbBjRhCsQ/SgwIpY53Lbp4gZAPjN66VX0kbuJ0Nocvp2dMl+9QkVq+9im8AEQEA +AYkCHwQYAQoACQUCWyB3EAIbDAAKCRDeT7zNgebHavfsEACp7nEkv7fYU9QgnGpP +/14yICPtnfBJzJ5Z35WkjSStBxd+Xdbnec1UlLqzGPPuXtLOcouRvmN6WXGPynB9 +ERd2Pga1yiexDZhVVR/vucwNq5pQv6LTlufMI1Nan/p6swwlsDWpVzPfGRaOMOt1 +1OaWeTA0q2ByDWQQV1qay09fs9dJsW6YBzFCKwNNaoXhM5zrmlQj80Ces7gMBJd+ +iivRVceRC+b2vAMvceX2FReS21o4S+mTKlTCvUNDbJe46a+rUEeHzBYN5qvbLedO +PQEXXUJLrjbj8/uA5gfW+Ib2azfpo7ezPnxYBNz4UC9CGF3JWFnYX7UHpwniB2pt +J9RrNiKL9Ohdpg1qNgJCAXzbrE31bqpaGawTFYMtRlEYHKDTs6jglX9cIVqDiz+y +smiMjVBM4G8PVwCvC0OfuCOqQrbieiSNfdBlcWJG5MbZqyXwQNiuRl5tdJPtB4Kx +NfSm3khyy1/xze0gOi4OwjznHXwW7606Uu6eAKKhWcQ7mHovEJSp5ziWb+/1PXe7 +f5XpIqrYxuMpghlA6Q72sON9zcmFEKW6y+3mRKtBzOsJXLKS0AWXi19zoFlq3Kzs +UiZuJIMHLPJK4sbOj5nMZE2163zUXz+gOVMeLqVxV/zlW6xnYu84taF4PbYIzmts +qG1EJoF+tteqemi1ZwYipD06wA== +=cA5d +-END PGP PUBLIC KEY BLOCK- - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24660][SHS] Show correct error pages when downloading logs
Repository: spark
Updated Branches:
refs/heads/master c04cb2d1b -> 776befbfd
[SPARK-24660][SHS] Show correct error pages when downloading logs
## What changes were proposed in this pull request?
SHS is showing bad errors when trying to download logs is not successful. This
may happen because the requested application doesn't exist or the user doesn't
have permissions for it, for instance.
The PR fixes the response when errors occur, so that they are displayed
properly.
## How was this patch tested?
manual tests
**Before the patch:**
1. Unauthorized user
2. Non-existing application
**After the patch**
1. Unauthorized user
2. Non-existing application
Author: Marco Gaido
Closes #21644 from mgaido91/SPARK-24660.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/776befbf
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/776befbf
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/776befbf
Branch: refs/heads/master
Commit: 776befbfd5b3c317a713d4fa3882cda6264db9ba
Parents: c04cb2d
Author: Marco Gaido
Authored: Wed Jun 27 14:26:08 2018 -0700
Committer: Marcelo Vanzin
Committed: Wed Jun 27 14:26:08 2018 -0700
--
.../spark/status/api/v1/ApiRootResource.scala | 30
.../status/api/v1/JacksonMessageWriter.scala| 5 +---
.../status/api/v1/OneApplicationResource.scala | 7 ++---
.../scala/org/apache/spark/ui/UIUtils.scala | 5
4 files changed, 13 insertions(+), 34 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/776befbf/core/src/main/scala/org/apache/spark/status/api/v1/ApiRootResource.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/status/api/v1/ApiRootResource.scala
b/core/src/main/scala/org/apache/spark/status/api/v1/ApiRootResource.scala
index d121068..84c2ad4 100644
--- a/core/src/main/scala/org/apache/spark/status/api/v1/ApiRootResource.scala
+++ b/core/src/main/scala/org/apache/spark/status/api/v1/ApiRootResource.scala
@@ -28,7 +28,7 @@ import org.glassfish.jersey.server.ServerProperties
import org.glassfish.jersey.servlet.ServletContainer
import org.apache.spark.SecurityManager
-import org.apache.spark.ui.SparkUI
+import org.apache.spark.ui.{SparkUI, UIUtils}
/**
* Main entry point for serving spark application metrics as json, using
JAX-RS.
@@ -148,38 +148,18 @@ private[v1] trait BaseAppResource extends
ApiRequestContext {
}
private[v1] class ForbiddenException(msg: String) extends
WebApplicationException(
- Response.status(Response.Status.FORBIDDEN).entity(msg).build())
+UIUtils.buildErrorResponse(Response.Status.FORBIDDEN, msg))
private[v1] class NotFoundException(msg: String) extends
WebApplicationException(
- new NoSuchElementException(msg),
-Response
- .status(Response.Status.NOT_FOUND)
- .entity(ErrorWrapper(msg))
- .build()
-)
+UIUtils.buildErrorResponse(Response.Status.NOT_FOUND, msg))
private[v1] class ServiceUnavailable(msg: String) extends
WebApplicationException(
- new ServiceUnavailableException(msg),
- Response
-.status(Response.Status.SERVICE_UNAVAILABLE)
-.entity(ErrorWrapper(msg))
-.build()
-)
+UIUtils.buildErrorResponse(Response.Status.SERVICE_UNAVAILABLE, msg))
private[v1] class BadParameterException(msg: String) extends
WebApplicationException(
- new IllegalArgumentException(msg),
- Response
-.status(Response.Status.BAD_REQUEST)
-.entity(ErrorWrapper(msg))
-.build()
-) {
+UIUtils.buildErrorResponse(Response.Status.BAD_REQUEST, msg)) {
def this(param: String, exp: String, actual: String) = {
this(raw"""Bad value for parameter "$param". Expected a $exp, got
"$actual)
}
}
-/**
- * Signal to JacksonMessageWriter to not convert the message into json (which
would result in an
- * extra set of quotes).
- */
-private[v1] case class ErrorWrapper(s: String)
http://git-wip-us.apache.org/repos/asf/spark/blob/776befbf/core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
spark git commit: [SPARK-24533] Typesafe rebranded to lightbend. Changing the build downloads path
Repository: spark
Updated Branches:
refs/heads/master 776befbfd -> 221d03acc
[SPARK-24533] Typesafe rebranded to lightbend. Changing the build downloads path
Typesafe has rebranded to lightbend. Just changing the downloads path to avoid
redirection
Tested by running build/mvn -DskipTests package
Author: Sanket Chintapalli
Closes #21636 from redsanket/SPARK-24533.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/221d03ac
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/221d03ac
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/221d03ac
Branch: refs/heads/master
Commit: 221d03acca19bdf7a2624a29c180c99f098205d8
Parents: 776befb
Author: Sanket Chintapalli
Authored: Wed Jun 27 14:37:19 2018 -0700
Committer: Marcelo Vanzin
Committed: Wed Jun 27 14:37:24 2018 -0700
--
build/mvn | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/221d03ac/build/mvn
--
diff --git a/build/mvn b/build/mvn
index 1405983..ae4276d 100755
--- a/build/mvn
+++ b/build/mvn
@@ -93,7 +93,7 @@ install_mvn() {
install_zinc() {
local zinc_path="zinc-0.3.15/bin/zinc"
[ ! -f "${_DIR}/${zinc_path}" ] && ZINC_INSTALL_FLAG=1
- local TYPESAFE_MIRROR=${TYPESAFE_MIRROR:-https://downloads.typesafe.com}
+ local TYPESAFE_MIRROR=${TYPESAFE_MIRROR:-https://downloads.lightbend.com}
install_app \
"${TYPESAFE_MIRROR}/zinc/0.3.15" \
@@ -109,7 +109,7 @@ install_scala() {
# determine the Scala version used in Spark
local scala_version=`grep "scala.version" "${_DIR}/../pom.xml" | head -n1 |
awk -F '[<>]' '{print $3}'`
local scala_bin="${_DIR}/scala-${scala_version}/bin/scala"
- local TYPESAFE_MIRROR=${TYPESAFE_MIRROR:-https://downloads.typesafe.com}
+ local TYPESAFE_MIRROR=${TYPESAFE_MIRROR:-https://downloads.lightbend.com}
install_app \
"${TYPESAFE_MIRROR}/scala/${scala_version}" \
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24645][SQL] Skip parsing when csvColumnPruning enabled and partitions scanned only
Repository: spark
Updated Branches:
refs/heads/master c5aa54d54 -> bd32b509a
[SPARK-24645][SQL] Skip parsing when csvColumnPruning enabled and partitions
scanned only
## What changes were proposed in this pull request?
In the master, when `csvColumnPruning`(implemented in [this
commit](https://github.com/apache/spark/commit/64fad0b519cf35b8c0a0dec18dd3df9488a5ed25#diff-d19881aceddcaa5c60620fdcda99b4c4))
enabled and partitions scanned only, it throws an exception below;
```
scala> val dir = "/tmp/spark-csv/csv"
scala> spark.range(10).selectExpr("id % 2 AS p",
"id").write.mode("overwrite").partitionBy("p").csv(dir)
scala> spark.read.csv(dir).selectExpr("sum(p)").collect()
18/06/25 13:12:51 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 5)
java.lang.NullPointerException
at
org.apache.spark.sql.execution.datasources.csv.UnivocityParser.org$apache$spark$sql$execution$datasources$csv$UnivocityParser$$convert(UnivocityParser.scala:197)
at
org.apache.spark.sql.execution.datasources.csv.UnivocityParser.parse(UnivocityParser.scala:190)
at
org.apache.spark.sql.execution.datasources.csv.UnivocityParser$$anonfun$5.apply(UnivocityParser.scala:309)
at
org.apache.spark.sql.execution.datasources.csv.UnivocityParser$$anonfun$5.apply(UnivocityParser.scala:309)
at
org.apache.spark.sql.execution.datasources.FailureSafeParser.parse(FailureSafeParser.scala:61)
...
```
This pr modified code to skip CSV parsing in the case.
## How was this patch tested?
Added tests in `CSVSuite`.
Author: Takeshi Yamamuro
Closes #21631 from maropu/SPARK-24645.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bd32b509
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bd32b509
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bd32b509
Branch: refs/heads/master
Commit: bd32b509a1728366494cba13f8f6612b7bd46ec0
Parents: c5aa54d
Author: Takeshi Yamamuro
Authored: Thu Jun 28 09:19:25 2018 +0800
Committer: hyukjinkwon
Committed: Thu Jun 28 09:19:25 2018 +0800
--
.../sql/execution/datasources/csv/UnivocityParser.scala | 10 +-
.../spark/sql/execution/datasources/csv/CSVSuite.scala| 10 ++
2 files changed, 19 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/bd32b509/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
index 5f7d569..aa545e1 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
@@ -183,11 +183,19 @@ class UnivocityParser(
}
}
+ private val doParse = if (schema.nonEmpty) {
+(input: String) => convert(tokenizer.parseLine(input))
+ } else {
+// If `columnPruning` enabled and partition attributes scanned only,
+// `schema` gets empty.
+(_: String) => InternalRow.empty
+ }
+
/**
* Parses a single CSV string and turns it into either one resulting row or
no row (if the
* the record is malformed).
*/
- def parse(input: String): InternalRow = convert(tokenizer.parseLine(input))
+ def parse(input: String): InternalRow = doParse(input)
private def convert(tokens: Array[String]): InternalRow = {
if (tokens.length != schema.length) {
http://git-wip-us.apache.org/repos/asf/spark/blob/bd32b509/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
index 365239d..84b91f6 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
@@ -1569,4 +1569,14 @@ class CSVSuite extends QueryTest with SharedSQLContext
with SQLTestUtils with Te
assert(testAppender2.events.asScala
.exists(msg => msg.getRenderedMessage.contains("CSV header does not
conform to the schema")))
}
+
+ test("SPARK-24645 skip parsing when columnPruning enabled and partitions
scanned only") {
+withSQLConf(SQLConf.CSV_PARSER_COLUMN_PRUNING.key -> "true") {
+ withTempPath { path =>
+val dir = path.getAbsolutePath
+spark.range(10).selectExpr("id % 2 AS p",
spark git commit: [SPARK-24206][SQL][FOLLOW-UP] Update DataSourceReadBenchmark benchmark results
Repository: spark
Updated Branches:
refs/heads/master bd32b509a -> 1c9acc243
[SPARK-24206][SQL][FOLLOW-UP] Update DataSourceReadBenchmark benchmark results
## What changes were proposed in this pull request?
This pr corrected the default configuration (`spark.master=local[1]`) for
benchmarks. Also, this updated performance results on the AWS `r3.xlarge`.
## How was this patch tested?
N/A
Author: Takeshi Yamamuro
Closes #21625 from maropu/FixDataSourceReadBenchmark.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1c9acc24
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1c9acc24
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1c9acc24
Branch: refs/heads/master
Commit: 1c9acc2438f9a97134ae5213a12112b2361fbb78
Parents: bd32b50
Author: Takeshi Yamamuro
Authored: Thu Jun 28 09:21:10 2018 +0800
Committer: hyukjinkwon
Committed: Thu Jun 28 09:21:10 2018 +0800
--
.../benchmark/DataSourceReadBenchmark.scala | 296 ++-
1 file changed, 152 insertions(+), 144 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1c9acc24/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
--
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
index fc6d8ab..8711f5a 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/DataSourceReadBenchmark.scala
@@ -39,9 +39,11 @@ import org.apache.spark.util.{Benchmark, Utils}
object DataSourceReadBenchmark {
val conf = new SparkConf()
.setAppName("DataSourceReadBenchmark")
-.setIfMissing("spark.master", "local[1]")
+// Since `spark.master` always exists, overrides this value
+.set("spark.master", "local[1]")
.setIfMissing("spark.driver.memory", "3g")
.setIfMissing("spark.executor.memory", "3g")
+.setIfMissing("spark.ui.enabled", "false")
val spark = SparkSession.builder.config(conf).getOrCreate()
@@ -154,73 +156,73 @@ object DataSourceReadBenchmark {
}
}
-
/*
-Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
+OpenJDK 64-Bit Server VM 1.8.0_171-b10 on Linux
4.14.33-51.37.amzn1.x86_64
+Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
SQL Single TINYINT Column Scan: Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-SQL CSV 15231 / 15267 1.0
968.3 1.0X
-SQL Json 8476 / 8498 1.9
538.9 1.8X
-SQL Parquet Vectorized 121 / 127130.0
7.7 125.9X
-SQL Parquet MR1515 / 1543 10.4
96.3 10.1X
-SQL ORC Vectorized 164 / 171 95.9
10.4 92.9X
-SQL ORC Vectorized with copy 228 / 234 69.0
14.5 66.8X
-SQL ORC MR1297 / 1309 12.1
82.5 11.7X
+SQL CSV 22964 / 23096 0.7
1460.0 1.0X
+SQL Json 8469 / 8593 1.9
538.4 2.7X
+SQL Parquet Vectorized 164 / 177 95.8
10.4 139.9X
+SQL Parquet MR1687 / 1706 9.3
107.2 13.6X
+SQL ORC Vectorized 191 / 197 82.3
12.2 120.2X
+SQL ORC Vectorized with copy 215 / 219 73.2
13.7 106.9X
+SQL ORC MR1392 / 1412 11.3
88.5 16.5X
SQL Single SMALLINT Column Scan: Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-SQL CSV 16344 / 16374 1.0
1039.1 1.0X
-SQL Json 8634 / 8648 1.8
548.9 1.9X
-SQL Parquet Vectorized 172 / 177 91.5
10.9 95.1X
-SQL Parquet MR1744 / 1746 9.0
110.9 9.4X
-SQL ORC
spark git commit: [SPARK-24603][SQL] Fix findTightestCommonType reference in comments
Repository: spark
Updated Branches:
refs/heads/master 1c9acc243 -> 6a97e8eb3
[SPARK-24603][SQL] Fix findTightestCommonType reference in comments
findTightestCommonTypeOfTwo has been renamed to findTightestCommonType
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests,
manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a pull
request.
Author: Fokko Driesprong
Closes #21597 from Fokko/fd-typo.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6a97e8eb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6a97e8eb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6a97e8eb
Branch: refs/heads/master
Commit: 6a97e8eb31da76fe5af912a6304c07b63735062f
Parents: 1c9acc2
Author: Fokko Driesprong
Authored: Thu Jun 28 09:59:00 2018 +0800
Committer: hyukjinkwon
Committed: Thu Jun 28 09:59:00 2018 +0800
--
.../spark/sql/execution/datasources/json/JsonInferSchema.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/6a97e8eb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
index f6edc7b..8e1b430 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
@@ -334,8 +334,8 @@ private[sql] object JsonInferSchema {
ArrayType(compatibleType(elementType1, elementType2), containsNull1
|| containsNull2)
// The case that given `DecimalType` is capable of given
`IntegralType` is handled in
-// `findTightestCommonTypeOfTwo`. Both cases below will be executed
only when
-// the given `DecimalType` is not capable of the given `IntegralType`.
+// `findTightestCommonType`. Both cases below will be executed only
when the given
+// `DecimalType` is not capable of the given `IntegralType`.
case (t1: IntegralType, t2: DecimalType) =>
compatibleType(DecimalType.forType(t1), t2)
case (t1: DecimalType, t2: IntegralType) =>
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24603][SQL] Fix findTightestCommonType reference in comments
Repository: spark
Updated Branches:
refs/heads/branch-2.2 47958270f -> a8537a5ab
[SPARK-24603][SQL] Fix findTightestCommonType reference in comments
findTightestCommonTypeOfTwo has been renamed to findTightestCommonType
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests,
manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a pull
request.
Author: Fokko Driesprong
Closes #21597 from Fokko/fd-typo.
(cherry picked from commit 6a97e8eb31da76fe5af912a6304c07b63735062f)
Signed-off-by: hyukjinkwon
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a8537a5a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a8537a5a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a8537a5a
Branch: refs/heads/branch-2.2
Commit: a8537a5ab876e5f25c491bf3f15f339a5a7c2b79
Parents: 4795827
Author: Fokko Driesprong
Authored: Thu Jun 28 09:59:00 2018 +0800
Committer: hyukjinkwon
Committed: Thu Jun 28 10:00:59 2018 +0800
--
.../spark/sql/execution/datasources/json/JsonInferSchema.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/a8537a5a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
index fb632cf..0987969 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
@@ -326,8 +326,8 @@ private[sql] object JsonInferSchema {
ArrayType(compatibleType(elementType1, elementType2), containsNull1
|| containsNull2)
// The case that given `DecimalType` is capable of given
`IntegralType` is handled in
-// `findTightestCommonTypeOfTwo`. Both cases below will be executed
only when
-// the given `DecimalType` is not capable of the given `IntegralType`.
+// `findTightestCommonType`. Both cases below will be executed only
when the given
+// `DecimalType` is not capable of the given `IntegralType`.
case (t1: IntegralType, t2: DecimalType) =>
compatibleType(DecimalType.forType(t1), t2)
case (t1: DecimalType, t2: IntegralType) =>
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24603][SQL] Fix findTightestCommonType reference in comments
Repository: spark
Updated Branches:
refs/heads/branch-2.3 6e1f5e018 -> 0f534d3da
[SPARK-24603][SQL] Fix findTightestCommonType reference in comments
findTightestCommonTypeOfTwo has been renamed to findTightestCommonType
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests,
manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a pull
request.
Author: Fokko Driesprong
Closes #21597 from Fokko/fd-typo.
(cherry picked from commit 6a97e8eb31da76fe5af912a6304c07b63735062f)
Signed-off-by: hyukjinkwon
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0f534d3d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0f534d3d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0f534d3d
Branch: refs/heads/branch-2.3
Commit: 0f534d3da284947f52b62154daaddfa77b42873c
Parents: 6e1f5e0
Author: Fokko Driesprong
Authored: Thu Jun 28 09:59:00 2018 +0800
Committer: hyukjinkwon
Committed: Thu Jun 28 10:00:38 2018 +0800
--
.../spark/sql/execution/datasources/json/JsonInferSchema.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/0f534d3d/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
index a270a64..1702fb6 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
@@ -329,8 +329,8 @@ private[sql] object JsonInferSchema {
ArrayType(compatibleType(elementType1, elementType2), containsNull1
|| containsNull2)
// The case that given `DecimalType` is capable of given
`IntegralType` is handled in
-// `findTightestCommonTypeOfTwo`. Both cases below will be executed
only when
-// the given `DecimalType` is not capable of the given `IntegralType`.
+// `findTightestCommonType`. Both cases below will be executed only
when the given
+// `DecimalType` is not capable of the given `IntegralType`.
case (t1: IntegralType, t2: DecimalType) =>
compatibleType(DecimalType.forType(t1), t2)
case (t1: DecimalType, t2: IntegralType) =>
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFileFormat
Repository: spark
Updated Branches:
refs/heads/master 221d03acc -> 893ea224c
[SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFileFormat
## What changes were proposed in this pull request?
This pr added code to verify a schema in Json/Orc/ParquetFileFormat along with
CSVFileFormat.
## How was this patch tested?
Added verification tests in `FileBasedDataSourceSuite` and
`HiveOrcSourceSuite`.
Author: Takeshi Yamamuro
Closes #21389 from maropu/SPARK-24204.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/893ea224
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/893ea224
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/893ea224
Branch: refs/heads/master
Commit: 893ea224cc738766be207c87f4b913fe8fea4c94
Parents: 221d03a
Author: Takeshi Yamamuro
Authored: Wed Jun 27 15:25:51 2018 -0700
Committer: Xiao Li
Committed: Wed Jun 27 15:25:51 2018 -0700
--
.../execution/datasources/DataSourceUtils.scala | 106 +
.../datasources/csv/CSVFileFormat.scala | 4 +-
.../execution/datasources/csv/CSVUtils.scala| 19 --
.../datasources/json/JsonFileFormat.scala | 4 +
.../datasources/orc/OrcFileFormat.scala | 4 +
.../datasources/parquet/ParquetFileFormat.scala | 3 +
.../spark/sql/FileBasedDataSourceSuite.scala| 213 ++-
.../execution/datasources/csv/CSVSuite.scala| 33 ---
.../spark/sql/hive/orc/OrcFileFormat.scala | 4 +
.../spark/sql/hive/orc/HiveOrcSourceSuite.scala | 49 -
10 files changed, 383 insertions(+), 56 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/893ea224/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
new file mode 100644
index 000..c534721
--- /dev/null
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import org.apache.spark.sql.execution.datasources.csv.CSVFileFormat
+import org.apache.spark.sql.execution.datasources.json.JsonFileFormat
+import org.apache.spark.sql.execution.datasources.orc.OrcFileFormat
+import org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat
+import org.apache.spark.sql.types._
+
+
+object DataSourceUtils {
+
+ /**
+ * Verify if the schema is supported in datasource in write path.
+ */
+ def verifyWriteSchema(format: FileFormat, schema: StructType): Unit = {
+verifySchema(format, schema, isReadPath = false)
+ }
+
+ /**
+ * Verify if the schema is supported in datasource in read path.
+ */
+ def verifyReadSchema(format: FileFormat, schema: StructType): Unit = {
+verifySchema(format, schema, isReadPath = true)
+ }
+
+ /**
+ * Verify if the schema is supported in datasource. This verification should
be done
+ * in a driver side, e.g., `prepareWrite`, `buildReader`, and
`buildReaderWithPartitionValues`
+ * in `FileFormat`.
+ *
+ * Unsupported data types of csv, json, orc, and parquet are as follows;
+ * csv -> R/W: Interval, Null, Array, Map, Struct
+ * json -> W: Interval
+ * orc -> W: Interval, Null
+ * parquet -> R/W: Interval, Null
+ */
+ private def verifySchema(format: FileFormat, schema: StructType, isReadPath:
Boolean): Unit = {
+def throwUnsupportedException(dataType: DataType): Unit = {
+ throw new UnsupportedOperationException(
+s"$format data source does not support ${dataType.simpleString} data
type.")
+}
+
+def verifyType(dataType: DataType): Unit = dataType match {
+ case BooleanType | ByteType | ShortType | IntegerType | LongType |
FloatType | DoubleType |
+ StringType | BinaryType | DateType |
spark git commit: [SPARK-24553][WEB-UI] http 302 fixes for href redirect
Repository: spark
Updated Branches:
refs/heads/master 893ea224c -> c5aa54d54
[SPARK-24553][WEB-UI] http 302 fixes for href redirect
## What changes were proposed in this pull request?
Updated URL/href links to include a '/' before '?id' to make links consistent
and avoid http 302 redirect errors within UI port 4040 tabs.
## How was this patch tested?
Built a runnable distribution and executed jobs. Validated that http 302
redirects are no longer encountered when clicking on links within UI port 4040
tabs.
Author: Steven Kallman
Author: Kallman, Steven
Closes #21600 from SJKallman/{Spark-24553}{WEB-UI}-redirect-href-fixes.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c5aa54d5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c5aa54d5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c5aa54d5
Branch: refs/heads/master
Commit: c5aa54d54b301555bad1ff0653df11293f0033ed
Parents: 893ea22
Author: Kallman, Steven
Authored: Wed Jun 27 15:36:59 2018 -0700
Committer: Xiao Li
Committed: Wed Jun 27 15:36:59 2018 -0700
--
core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala | 2 +-
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala | 2 +-
core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala| 2 +-
.../org/apache/spark/sql/execution/ui/AllExecutionsPage.scala| 4 ++--
.../scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala | 2 +-
.../apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala | 4 ++--
.../spark/sql/hive/thriftserver/ui/ThriftServerSessionPage.scala | 2 +-
.../src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala | 2 +-
8 files changed, 10 insertions(+), 10 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/c5aa54d5/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
--
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
index 178d2c8..90e9a7a 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala
@@ -464,7 +464,7 @@ private[ui] class JobDataSource(
val jobDescription = UIUtils.makeDescription(lastStageDescription,
basePath, plainText = false)
-val detailUrl = "%s/jobs/job?id=%s".format(basePath, jobData.jobId)
+val detailUrl = "%s/jobs/job/?id=%s".format(basePath, jobData.jobId)
new JobTableRowData(
jobData,
http://git-wip-us.apache.org/repos/asf/spark/blob/c5aa54d5/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
--
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
index d4e6a7b..55eb989 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
@@ -282,7 +282,7 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
val _taskTable = new TaskPagedTable(
stageData,
UIUtils.prependBaseUri(request, parent.basePath) +
- s"/stages/stage?id=${stageId}=${stageAttemptId}",
+ s"/stages/stage/?id=${stageId}=${stageAttemptId}",
currentTime,
pageSize = taskPageSize,
sortColumn = taskSortColumn,
http://git-wip-us.apache.org/repos/asf/spark/blob/c5aa54d5/core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
--
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
b/core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
index 56e4d68..d01acda 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala
@@ -370,7 +370,7 @@ private[ui] class StagePagedTable(
Seq.empty
}
-val nameLinkUri =
s"$basePathUri/stages/stage?id=${s.stageId}=${s.attemptId}"
+val nameLinkUri =
s"$basePathUri/stages/stage/?id=${s.stageId}=${s.attemptId}"
val nameLink = {s.name}
val cachedRddInfos = store.rddList().filter { rdd =>
s.rddIds.contains(rdd.id) }
http://git-wip-us.apache.org/repos/asf/spark/blob/c5aa54d5/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala
--
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala
index bf46bc4..a7a24ac 100644
---
svn commit: r27779 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_27_16_01-c5aa54d-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Wed Jun 27 23:15:46 2018 New Revision: 27779 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_27_16_01-c5aa54d docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r27756 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_27_00_01-9a76f23-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Wed Jun 27 07:17:22 2018 New Revision: 27756 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_27_00_01-9a76f23 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-21335][DOC] doc changes for disallowed un-aliased subquery use case
Repository: spark Updated Branches: refs/heads/master 9a76f23c6 -> a1a64e358 [SPARK-21335][DOC] doc changes for disallowed un-aliased subquery use case ## What changes were proposed in this pull request? Document a change for un-aliased subquery use case, to address the last question in PR #18559: https://github.com/apache/spark/pull/18559#issuecomment-316884858 (Please fill in changes proposed in this fix) ## How was this patch tested? it does not affect tests. Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Yuexin Zhang Closes #21647 from cnZach/doc_change_for_SPARK-20690_SPARK-21335. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a1a64e35 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a1a64e35 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a1a64e35 Branch: refs/heads/master Commit: a1a64e3583cfa451b4d0d2361c1da2972a5e Parents: 9a76f23 Author: Yuexin Zhang Authored: Wed Jun 27 16:05:36 2018 +0800 Committer: Wenchen Fan Committed: Wed Jun 27 16:05:36 2018 +0800 -- docs/sql-programming-guide.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/spark/blob/a1a64e35/docs/sql-programming-guide.md -- diff --git a/docs/sql-programming-guide.md b/docs/sql-programming-guide.md index 7c4ef41..cd7329b 100644 --- a/docs/sql-programming-guide.md +++ b/docs/sql-programming-guide.md @@ -2017,6 +2017,7 @@ working with timestamps in `pandas_udf`s to get the best performance, see - Literal values used in SQL operations are converted to DECIMAL with the exact precision and scale needed by them. - The configuration `spark.sql.decimalOperations.allowPrecisionLoss` has been introduced. It defaults to `true`, which means the new behavior described here; if set to `false`, Spark uses previous rules, ie. it doesn't adjust the needed scale to represent the values and it returns NULL if an exact representation of the value is not possible. - In PySpark, `df.replace` does not allow to omit `value` when `to_replace` is not a dictionary. Previously, `value` could be omitted in the other cases and had `None` by default, which is counterintuitive and error-prone. + - Un-aliased subquery's semantic has not been well defined with confusing behaviors. Since Spark 2.3, we invalidate such confusing cases, for example: `SELECT v.i from (SELECT i FROM v)`, Spark will throw an analysis exception in this case because users should not be able to use the qualifier inside a subquery. See [SPARK-20690](https://issues.apache.org/jira/browse/SPARK-20690) and [SPARK-21335](https://issues.apache.org/jira/browse/SPARK-21335) for more details. ## Upgrading From Spark SQL 2.1 to 2.2 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r27763 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_27_04_02-a1a64e3-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Wed Jun 27 11:21:26 2018 New Revision: 27763 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_27_04_02-a1a64e3 docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r27784 - in /dev/spark/2.3.2-SNAPSHOT-2018_06_27_22_01-0f534d3-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Thu Jun 28 05:16:02 2018 New Revision: 27784 Log: Apache Spark 2.3.2-SNAPSHOT-2018_06_27_22_01-0f534d3 docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r27783 - in /dev/spark/2.4.0-SNAPSHOT-2018_06_27_20_01-6a97e8e-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Thu Jun 28 03:15:54 2018 New Revision: 27783 Log: Apache Spark 2.4.0-SNAPSHOT-2018_06_27_20_01-6a97e8e docs [This commit notification would consist of 1468 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
