ot;option" -> true)
res0: scala.collection.immutable.Map[String,Boolean] = Map(option -> true)
```
## How was this patch tested?
Added a test for checking which option is propagated to a data source in
`load()`.
Closes #22413 from MaxGekk/session-options.
Lead-authored-by: Maxim Gekk
Co-authored-by: D
minutes.
Closes #22427 from dongjoon-hyun/SPARK-25438.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fefaa3c3
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree
minutes.
Closes #22427 from dongjoon-hyun/SPARK-25438.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit fefaa3c30df2c56046370081cb51bfe68d26976b)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip
Repository: spark
Updated Branches:
refs/heads/master e06da95cd -> fefaa3c30
http://git-wip-us.apache.org/repos/asf/spark/blob/fefaa3c3/sql/core/benchmarks/FilterPushdownBenchmark-results.txt
--
diff --git a/sql/core/benchmarks
Repository: spark
Updated Branches:
refs/heads/branch-2.4 ae2ca0e5d -> b40e5feec
http://git-wip-us.apache.org/repos/asf/spark/blob/b40e5fee/sql/core/benchmarks/FilterPushdownBenchmark-results.txt
--
diff --git a/sql/core/benchm
tested?
unit tests
Closes #22435 from wangyum/SPARK-25423.
Authored-by: Yuming Wang
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4b9542e3
Tree: http://git-wip-us.apache.org/repos/asf/spark/
ime is spent in the code generation/compilation process.
## How was this patch tested?
added UTs
Closes #22395 from mgaido91/SPARK-16323.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/a
t: http://git-wip-us.apache.org/repos/asf/spark/commit/cb1b55cf
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cb1b55cf
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cb1b55cf
Branch: refs/heads/master
Commit: cb1b55cf771018f1560f6b173cdd7c6ca8061bc7
Parents: a71f6a1
Author:
61bc7)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9fefb47f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9fefb47f
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9fefb47f
tra options in DataSource
V2. Extra options are more specific and set via `.option()`, and should
overwrite more generic session options.
## How was this patch tested?
Added tests for read and write paths.
Closes #22474 from MaxGekk/session-options-2.4.
Authored-by: Maxim Gekk
Signed-off-by: Dongj
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9bf04d85
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9bf04d85
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9bf04d85
Branc
ult:
```sh
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
org.apache.spark.sql.execution.columnar.compression.CompressionSchemeBenchmark"
```
## How was this patch tested?
manual tests
Closes #22486 from wangyum/SPARK-25478.
Lead-authored-by: Yuming Wang
Co-authored
hored-by: hyukjinkwon
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ec384284
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ec384284
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ec384284
fix**
## How was this patch tested?
Manually tested
Closes #22525 from shahidki31/SparkUI.
Authored-by: Shahid
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/sp
fix**
## How was this patch tested?
Manually tested
Closes #22525 from shahidki31/SparkUI.
Authored-by: Shahid
Signed-off-by: Dongjoon Hyun
(cherry picked from com
fix**
## How was this patch tested?
Manually tested
Closes #22525 from shahidki31/SparkUI.
Authored-by: Shahid
Signed-off-by: Dongjoon Hyun
(cherry picked from com
ult:
```
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
org.apache.spark.sql.execution.benchmark.SortBenchmark"
```
## How was this patch tested?
manual tests
Closes #22495 from yucai/SPARK-25486.
Authored-by: yucai
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spar
ssion(spark)
super.withSQLConf(pairs: _*)(f)
}
```
## How was this patch tested?
Pass the Jenkins with the existing tests.
Closes #22548 from dongjoon-hyun/SPARK-25534.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: htt
tra options in DataSource
V2. Extra options are more specific and set via `.option()`, and should
overwrite more generic session options.
## How was this patch tested?
Added tests for read and write paths.
Closes #22489 from MaxGekk/session-options-2.3.
Authored-by: Maxim Gekk
Signed-off-by: Dongj
is computed only once.
## How was this patch tested?
Removed a test case that's no longer valid.
Closes #22521 from rxin/SPARK-24519.
Authored-by: Reynold Xin
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/a
is computed only once.
## How was this patch tested?
Removed a test case that's no longer valid.
Closes #22521 from rxin/SPARK-24519.
Authored-by: Reynold Xin
Signed-off-by: Dongjoon Hyun
(cherry picked from commit e702fb1d5218d062fcb8e618b92dad7958eb4062)
Signed-off-by: Dongjoon Hyun
ult:
```
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
org.apache.spark.sql.execution.vectorized.ColumnarBatchBenchmark"
```
## How was this patch tested?
manual tests
Closes #22490 from yucai/SPARK-25481.
Lead-authored-by: yucai
Co-authored-by: Yucai Yu
Co-authored-by:
rom
'task.metrics.inputMetrics.recordsRead' to
'task.metrics.outputMetrics.recordsWritten'.
This bug was introduced in SPARK-22190.
https://github.com/apache/spark/pull/19426
## How was this patch tested?
Existing tests
Closes #22555 from shahidki31/SPARK-25536.
Authored-by: Shahid
Signed-off-by: D
rom
'task.metrics.inputMetrics.recordsRead' to
'task.metrics.outputMetrics.recordsWritten'.
This bug was introduced in SPARK-22190.
https://github.com/apache/spark/pull/19426
## How was this patch tested?
Existing tests
Closes #22555 from shahidki31/SPARK-25536.
Authored-by: Shahid
Signe
rom
'task.metrics.inputMetrics.recordsRead' to
'task.metrics.outputMetrics.recordsWritten'.
This bug was introduced in SPARK-22190.
https://github.com/apache/spark/pull/19426
## How was this patch tested?
Existing tests
Closes #22555 from shahidki31/SPARK-25536.
Authored-by: Shahid
Signe
ult:
```
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "catalyst/test:runMain
org.apache.spark.sql.UnsafeProjectionBenchmark"
```
## How was this patch tested?
manual test
Closes #22493 from yucai/SPARK-25485.
Lead-authored-by: yucai
Co-authored-by: Yucai Yu
Co-authored-by: Dongjoon Hyu
o breaks
tests that use RDDInfo outside of the scope of a SparkContext.
Since this is not a performance sensitive area, there's no advantage
in caching the config value.
Closes #22558 from vanzin/SPARK-25546.
Authored-by: Marcelo Vanzin
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-
It also breaks
tests that use RDDInfo outside of the scope of a SparkContext.
Since this is not a performance sensitive area, there's no advantage
in caching the config value.
Closes #22558 from vanzin/SPARK-25546.
Authored-by: Marcelo Vanzin
Signed-off-by: Dongjoon Hyun
(cherry picke
Co-authored-by: Christopher Zhao
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3b7395fe
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3b7395fe
Diff: h
ual via `SKIP_API=1 jekyll build`.
Closes #22585 from dongjoon-hyun/SPARK-23285.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0b33f086
Tree: http://git-wip-us.apache.
ted?
Manual via `SKIP_API=1 jekyll build`.
Closes #22585 from dongjoon-hyun/SPARK-23285.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 0b33f08683a41f6f3a6ec02c327010c0722cc1d1)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/r
ted?
Existing tests.
Closes #22569 from viirya/SPARK-25542.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b7d80349
Tree: http://git-wip-us.apache.org/repos/asf/spark/t
ted?
Existing tests.
Closes #22569 from viirya/SPARK-25542.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Dongjoon Hyun
(cherry picked from commit b7d80349b0e367d78cab238e62c2ec353f0f12b3)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-
sai/removeUnsupportedPredicatesInParquet.
Lead-authored-by: DB Tsai
Co-authored-by: Dongjoon Hyun
Co-authored-by: DB Tsai
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5d726b86
Tree: http://git-
ult:
```
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "hive/test:runMain
org.apache.spark.sql.hive.orc.OrcReadBenchmark"
```
## How was this patch tested?
manual tests
Closes #22580 from yucai/SPARK-25508.
Lead-authored-by: yucai
Co-authored-by: Yucai Yu
Co-authored-by: Dongjoon Hyun
Signed-off-by:
-by: Prashant Sharma
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4da541a5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4da541a5
Diff: http://git-wip-us.apache.org/repos/asf/spark/d
red-by: Prashant Sharma
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 4da541a5d23b039eb549dd849cf121bdc8676e59)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c886f050
Tree: http://git-
How was this patch tested?
manual tests
Closes #22484 from wangyum/SPARK-25476.
Lead-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/
Repository: spark-website
Updated Branches:
refs/heads/asf-site 8b7444182 -> 4ea5a5d8f
Add a section to update test suites
Author: Dongjoon Hyun
Closes #151 from dongjoon-hyun/test_suite.
Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo
Commit: http://git-
8-9689-e9b39882a7b5.png)
Closes #22601 from shahidki31/historyConf.
Authored-by: Shahid
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/71876633
Tree: http://git-wip-us.apache.org/repos/asf/s
-11e8-9689-e9b39882a7b5.png)
Closes #22601 from shahidki31/historyConf.
Authored-by: Shahid
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 71876633f3af706408355b5fb561b58dbc593360)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http:
339
## How was this patch tested?
Unit test.
Closes #22599 from gengliangwang/renameBenchmarkSuite.
Authored-by: Gengliang Wang
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7b4e94f1
Tree: http://
ses before merging
Related Job:
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-branch-2.2-lint/913/console
Closes #22596 from samdvr/SPARK-25576.
Lead-authored-by: Sam Davarnia
Co-authored-by: Sam Davarnia <>
Co-authored-by: Dongjoon Hyun
Co-authored-by: Sam
#22613 from shahidki31/SPARK-25583.
Authored-by: Shahid
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7102aeeb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7102aeeb
Diff: http://git-
ong time. For Parquet data source, it's expected to
be added with next Parquet release update.
## How was this patch tested?
Manual.
```scala
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
org.apache.spark.sql.execution.benchmark.BloomFilterBenchmark"
```
Closes #2
`sh
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
org.apache.spark.sql.execution.benchmark.UnsafeArrayDataBenchmark"
```
## How was this patch tested?
manual tests
Closes #22491 from wangyum/SPARK-25483.
Lead-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
is more efficient later.
## How was this patch tested?
Running the test attached in the JIRA + added UT
Closes #22602 from mgaido91/SPARK-25582.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.
ink is more efficient later.
## How was this patch tested?
Running the test attached in the JIRA + added UT
Closes #22602 from mgaido91/SPARK-25582.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
(cherry picked from commit d7ae36a810bfcbedfe7360eb2cdbbc3ca970e4d0)
Signed-off-by: Dongj
`sh
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
org.apache.spark.sql.DatasetBenchmark"
```
## How was this patch tested?
manual tests
Closes #22488 from wangyum/SPARK-25479.
Lead-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: h
dev build.
Tested on local dev build.
Closes #22634 from vanzin/SPARK-25646.
Authored-by: Marcelo Vanzin
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/58287a39
Tree: http://git-wip-us.apache.
n a dev build.
Tested on local dev build.
Closes #22634 from vanzin/SPARK-25646.
Authored-by: Marcelo Vanzin
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 58287a39864db463eeef17d1152d664be021d9ef)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/r
`sh
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain
org.apache.spark.sql.execution.benchmark.MiscBenchmark"
```
## How was this patch tested?
manual tests
Closes #22500 from wangyum/SPARK-25488.
Lead-authored-by: Yuming Wang
Co-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
ds.writeStream.foreachBatch((_, _) =>
{}).partitionBy("value").start()
[error] ^
```
## How was this patch tested?
Manual.
Since this failure occurs in Scala-2.12 profile and test cases, Jenkins will
not test this. We need to build with Scala-2.12 and run the tests
ds.writeStream.foreachBatch((_, _) =>
{}).partitionBy("value").start()
[error] ^
```
## How was this patch tested?
Manual.
Since this failure occurs in Scala-2.12 profile and test cases, Jenkins will
not test this. We need to build with Scala-2.12 and run the tests
ume
more memory. This PR adds the conversion to the latter case and decreases
memory consumption.
## How was this patch tested?
Added unit test.
Closes #22603 from peter-toth/SPARK-25062.
Authored-by: Peter Toth
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/sp
catalyst/test:runMain
org.apache.spark.sql.HashByteArrayBenchmark"
```
## How was this patch tested?
manual tests
Closes #22652 from wangyum/SPARK-25658.
Lead-authored-by: Yuming Wang
Co-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: h
:runMain
org.apache.spark.sql.HashBenchmark"
```
## How was this patch tested?
manual tests
Closes #22651 from wangyum/SPARK-25657.
Lead-authored-by: Yuming Wang
Co-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/a
ing changes.
cc: WeichenXu123
## How was this patch tested?
Existing unit tests.
Closes #22618 from mengxr/SPARK-25321.master.
Authored-by: WeichenXu
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/com
abble.com/Possible-bug-in-DatasourceV2-td25343.html
## How was this patch tested?
Unit test and manual tests.
Closes #22688 from HyukjinKwon/append-revert-2.
Authored-by: hyukjinkwon
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wi
:runMain
org.apache.spark.sql.execution.benchmark.JoinBenchmark"
```
## How was this patch tested?
manual tests
Closes #22661 from wangyum/SPARK-25664.
Lead-authored-by: Yuming Wang
Co-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: h
nts)
spark.history.fs.updateIntervalHow often to reload log data from storage
(in seconds, default: 10)
```
## How was this patch tested?
Manual test
Closes #22699 from gengliangwang/refactorSHSUsage.
Authored-by: Gengliang Wang
Signed-off-by: Dongj
" contained "123" (SaveIntoDataSourceCommandSuite.scala:42)
```
## How was this patch tested?
Pass the Jenkins with the updated test case
Closes #22716 from dongjoon-hyun/SPARK-25726.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/a
" contained "123" (SaveIntoDataSourceCommandSuite.scala:42)
```
## How was this patch tested?
Pass the Jenkins with the updated test case
Closes #22716 from dongjoon-hyun/SPARK-25726.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 6bbceb9fefe815d1
" contained "123" (SaveIntoDataSourceCommandSuite.scala:42)
```
## How was this patch tested?
Pass the Jenkins with the updated test case
Closes #22716 from dongjoon-hyun/SPARK-25726.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 6bbceb9fefe815d1
" contained "123" (SaveIntoDataSourceCommandSuite.scala:42)
```
## How was this patch tested?
Pass the Jenkins with the updated test case
Closes #22716 from dongjoon-hyun/SPARK-25726.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 6bbceb9fefe815d1
9)
at
org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.scala:604)
at
org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:590)
```
## How was this patch tested?
Added a test
Closes #22715 from gatorsmile/copyArgs1.
Authored-by: gatorsmile
Signed-off-by: Dongjoon Hyun
9)
at
org.apache.spark.sql.catalyst.trees.TreeNode.jsonValue(TreeNode.scala:604)
at
org.apache.spark.sql.catalyst.trees.TreeNode.toJSON(TreeNode.scala:590)
```
## How was this patch tested?
Added a test
Closes #22715 from gatorsmile/copyArgs1.
Authored-by: gatorsmile
Signed-off-by: Dongjoon
st/resources/TestUDTF.jar;
ADD JAR /Users/yumwang/spark/sql/hive/src/test/resources/TestUDTF.jar
spark-sql>
```
Closes #22747 from wangyum/AddJarCommand.
Authored-by: Yuming Wang
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apach
of `ClusteringEvaluator` in order to evaluate the
clustering.
## How was this patch tested?
NA
Closes #22756 from mgaido91/SPARK-25758.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/re
ion of `ClusteringEvaluator` in order to evaluate the
clustering.
## How was this patch tested?
NA
Closes #22756 from mgaido91/SPARK-25758.
Authored-by: Marco Gaido
Signed-off-by: Dongjoon Hyun
(cherry picked from commit c2962546d9a5900a5628a31b83d2c4b22c3a7936)
Signed-off-by: Dongjoon H
the java bean.
## How was this patch tested?
Added a test case.
Built complete project on travis.
dongjoon-hyun cloud-fan
Closes #22768 from vofque/SPARK-21402-2.2.
Lead-authored-by: Vladimir Kuriatkov
Co-authored-by: Vladimir Kuriatkov
Signed-off-by: Dongjoon Hyun
Project: http://git-
ark SQL 2.4 to 3.0` when
merging to 2.4
## How was this patch tested?
N/A
Closes #22769 from gatorsmile/test2.4.
Authored-by: gatorsmile
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fd5b2
the java bean.
## How was this patch tested?
Added a test case.
Built complete project on travis.
dongjoon-hyun cloud-fan
Closes #22767 from vofque/SPARK-21402-2.3.
Authored-by: Vladimir Kuriatkov
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: h
ng)
Closes #22263 from wangyum/SPARK-25269.
Authored-by: Yuming Wang
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9ad0f6ea
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9ad0f6
describe Java 8 only.
## How was this patch tested?
N/A (This is a document only change.)
Closes #22781 from dongjoon-hyun/SPARK-JDK-DOC.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/r
ment
to describe Java 8 only.
## How was this patch tested?
N/A (This is a document only change.)
Closes #22781 from dongjoon-hyun/SPARK-JDK-DOC.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
(cherry picked from commit fc9ba9dcc6ad47fbd05f093b94e7e1358d5f)
Signed-off-by: Dongj
s fixed error relates to `[SPARK-12995][GraphX] Remove deprecate APIs from
Pregel`
## How was this patch tested?
N/A
Closes #22780 from WeichenXu123/minor_doc_update1.
Authored-by: WeichenXu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 3b4f35f568eb3844d2a789c8a409bc705477df6b)
Signed-off
or relates to `[SPARK-12995][GraphX] Remove deprecate APIs from
Pregel`
## How was this patch tested?
N/A
Closes #22780 from WeichenXu123/minor_doc_update1.
Authored-by: WeichenXu
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-
s fixed error relates to `[SPARK-12995][GraphX] Remove deprecate APIs from
Pregel`
## How was this patch tested?
N/A
Closes #22780 from WeichenXu123/minor_doc_update1.
Authored-by: WeichenXu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 3b4f35f568eb3844d2a789c8a409bc705477df6b)
Signed-off
s fixed error relates to `[SPARK-12995][GraphX] Remove deprecate APIs from
Pregel`
## How was this patch tested?
N/A
Closes #22780 from WeichenXu123/minor_doc_update1.
Authored-by: WeichenXu
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 3b4f35f568eb3844d2a789c8a409bc705477df6b)
Signed-off
quot;sql/test:runMain
org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark"
```
## How was this patch tested?
manual tests
Closes #22501 from wangyum/SPARK-25492.
Lead-authored-by: Yuming Wang
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apa
ScanExec`
## How was this patch tested?
existing tests
Closes #22750 from cloud-fan/minor.
Authored-by: Wenchen Fan
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ab5752cb
Tree: http:/
ested?
Manual.
Closes #22776 from HyukjinKwon/SPARK-25779.
Authored-by: hyukjinkwon
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b8c6ba9e
Tree: http://git-wip-us.apache.org/repos/asf/spark/t
2GB
to merge everything, which will create OOM errors.
And we can avoid this issue by upgrade the netty.
https://github.com/netty/netty/pull/8038
## How was this patch tested?
Manual tests in some spark jobs.
Closes #22765 from lipzhu/SPARK-25757.
Authored-by: Zhu, Lipeng
Signed-off-by:
.df("examples/src/main/resources/people.csv", "csv", sep=";",
> inferSchema=T, header=T)
> namesAndAges <- select(df, "name", "age")
```
Closes #22791 from dongjoon-hyun/SPARK-25795.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon H
t;- read.df("examples/src/main/resources/people.csv", "csv", sep=";",
> inferSchema=T, header=T)
> namesAndAges <- select(df, "name", "age")
```
Closes #22791 from dongjoon-hyun/SPARK-25795.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon
t;- read.df("examples/src/main/resources/people.csv", "csv", sep=";",
> inferSchema=T, header=T)
> namesAndAges <- select(df, "name", "age")
```
Closes #22791 from dongjoon-hyun/SPARK-25795.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon
rom gengliangwang/fixPagedTableSuite.
Authored-by: Gengliang Wang
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/65a8d1b8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/65a8d1b8
Diff: http://
rce specific
options to the underlying data source. According to [the review
comment](https://github.com/apache/spark/pull/22622#discussion_r222911529),
this PR aims to add more detailed information and examples
## How was this patch tested?
Manual.
Closes #22801 from dongjoon-hyun/SPARK-25
;
```
Closes #22804 from peter-toth/SPARK-25665.
Lead-authored-by: Peter Toth
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ccd07b73
Tree: http://git-wip-us.apa
nce it's not supported in
ORC 1.5.2.
## How was this patch tested?
Manual.
Closes #22839 from dongjoon-hyun/SPARK-25656-2.4.
Authored-by: Dongjoon Hyun
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/a
e. (Personally, I did before.)
This PR aims to recover that script to work in any way. This doesn't aim for
source artifacts to reproduce the compiled artifacts.
## How was this patch tested?
Manual.
```
$ rm LICENSE-binary
$ dev/make-distribution.sh
```
Closes #22840 from dongjoon-hyun/SP
e. (Personally, I did before.)
This PR aims to recover that script to work in any way. This doesn't aim for
source artifacts to reproduce the compiled artifacts.
## How was this patch tested?
Manual.
```
$ rm LICENSE-binary
$ dev/make-distribution.sh
```
Closes #22840 from dongjoon-hyun/SPARK-2
g manual
[reverting](https://github.com/dongjoon-hyun/spark/commit/5d744499667fcd08825bca0ac6d5d90d6e110ebc#diff-79dd276be45ede6f34e24ad7005b0a7cR87).
This doens't cause test failure because the difference is inside `comments`
and blank lines. We had better fix this minor issue before RC5
1 jekyll serve --watch
```
Closes #22851 from seancxmao/SPARK-25797-2.3.
Authored-by: seancxmao
Signed-off-by: Dongjoon Hyun
(cherry picked from commit 3e0160bacfbe4597f15ca410ca832617cdeeddca)
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: ht
1 jekyll serve --watch
```
Closes #22851 from seancxmao/SPARK-25797-2.3.
Authored-by: seancxmao
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3e0160ba
Tree: http://git-wip-us.apache.org/repos/a
:runMain
org.apache.spark.sql.execution.datasources.csv.CSVBenchmark"`
## How was this patch tested?
manual tests
Closes #22845 from heary-cao/CSVBenchmarks.
Authored-by: caoxuewen
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apa
locally.
```
SKIP_API=1 jekyll serve
```
Closes #22868 from seancxmao/SPARK-25833.
Authored-by: seancxmao
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/49bea5a7
Tree: http://git-wip-us.apache.org/r
wal/SPARK-25618.
Authored-by: Dilip Biswal
Signed-off-by: Dongjoon Hyun
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/34c3bc9f
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/34c3bc9f
Diff: http://git-wip-us.apache.
;avro/test:runMain
org.apache.spark.sql.execution.benchmark.AvroWriteBenchmark"
```
## How was this patch tested?
manual tests
Closes #22861 from yucai/BuiltInDataSourceWriteBenchmark.
Lead-authored-by: yucai
Co-authored-by: Yucai Yu
Co-authored-by: Dongjoon Hyun
Signed-off-by: Dongjoo
1 - 100 of 8596 matches
Mail list logo
