GitHub user roxchkplusony opened a pull request:
https://github.com/apache/spark/pull/3502
[BRANCH-1.1][SPARK-4626] Kill a task only if the executorId is (still)
registered with the scheduler
v1.1 backport for #3483
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Paxata/spark bugfix/4626-1.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3502.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3502
----
commit 90f8f3eed026e9c4f1a4b1952e284558c0e3fd23
Author: chutium <[email protected]>
Date: 2014-08-27T20:13:04Z
[SPARK-3138][SQL] sqlContext.parquetFile should be able to take a single
file as parameter
```if (!fs.getFileStatus(path).isDir) throw Exception``` make no sense
after this commit #1370
be careful if someone is working on SPARK-2551, make sure the new change
passes test case ```test("Read a parquet file instead of a directory")```
Author: chutium <[email protected]>
Closes #2044 from chutium/parquet-singlefile and squashes the following
commits:
4ae477f [chutium] [SPARK-3138][SQL] sqlContext.parquetFile should be able
to take a single file as parameter
(cherry picked from commit 48f42781dedecd38ddcb2dcf67dead92bb4318f5)
Signed-off-by: Michael Armbrust <[email protected]>
commit 3cb4e1718f40a18e3d19a33fd627960687bbcb6c
Author: Vida Ha <[email protected]>
Date: 2014-08-27T21:26:06Z
Spark-3213 Fixes issue with spark-ec2 not detecting slaves created with
"Launch More like this"
... copy the spark_cluster_tag from a spot instance requests over to the
instances.
Author: Vida Ha <[email protected]>
Closes #2163 from vidaha/vida/spark-3213 and squashes the following commits:
5070a70 [Vida Ha] Spark-3214 Fix issue with spark-ec2 not detecting slaves
created with 'Launch More Like This' and using Spot Requests
(cherry picked from commit 7faf755ae4f0cf510048e432340260a6e609066d)
Signed-off-by: Josh Rosen <[email protected]>
commit c1ffa3e4cdfbd1f84b5c8d8de5d0fb958a19e211
Author: Andrew Or <[email protected]>
Date: 2014-08-27T21:46:56Z
[SPARK-3243] Don't use stale spark-driver.* system properties
If we set both `spark.driver.extraClassPath` and `--driver-class-path`,
then the latter correctly overrides the former. However, the value of the
system property `spark.driver.extraClassPath` still uses the former, which is
actually not added to the class path. This may cause some confusion...
Of course, this also affects other options (i.e. java options, library
path, memory...).
Author: Andrew Or <[email protected]>
Closes #2154 from andrewor14/driver-submit-configs-fix and squashes the
following commits:
17ec6fc [Andrew Or] Fix tests
0140836 [Andrew Or] Don't forget spark.driver.memory
e39d20f [Andrew Or] Also set spark.driver.extra* configs in client mode
(cherry picked from commit 63a053ab140d7bf605e8c5b7fb5a7bd52aca29b2)
Signed-off-by: Patrick Wendell <[email protected]>
commit b3d763b0b7fc6345dac5d222414f902e4afdee13
Author: viirya <[email protected]>
Date: 2014-08-27T21:55:05Z
[SPARK-3252][SQL] Add missing condition for test
According to the text message, both relations should be tested. So add the
missing condition.
Author: viirya <[email protected]>
Closes #2159 from viirya/fix_test and squashes the following commits:
b1c0f52 [viirya] add missing condition.
(cherry picked from commit 28d41d627919fcb196d9d31bad65d664770bee67)
Signed-off-by: Michael Armbrust <[email protected]>
commit 77116875f4184e0a637d9d7fd5b1dfeaabe0c9d3
Author: Aaron Davidson <[email protected]>
Date: 2014-08-27T22:05:47Z
[SQL] [SPARK-3236] Reading Parquet tables from Metastore mangles location
Currently we do `relation.hiveQlTable.getDataLocation.getPath`, which
returns the path-part of the URI (e.g., "s3n://my-bucket/my-path" =>
"/my-path"). We should do `relation.hiveQlTable.getDataLocation.toString`
instead, as a URI's toString returns a faithful representation of the full URI,
which can later be passed into a Hadoop Path.
Author: Aaron Davidson <[email protected]>
Closes #2150 from aarondav/parquet-location and squashes the following
commits:
459f72c [Aaron Davidson] [SQL] [SPARK-3236] Reading Parquet tables from
Metastore mangles location
(cherry picked from commit cc275f4b7910f6d0ad266a43bac2fdae58e9739e)
Signed-off-by: Michael Armbrust <[email protected]>
commit 5ea260ebd1acbbe9705849a16ee67758e33c65b0
Author: luogankun <[email protected]>
Date: 2014-08-27T22:08:22Z
[SPARK-3065][SQL] Add locale setting to fix results do not match for
udf_unix_timestamp format "yyyy MMM dd h:mm:ss a" run with not
"America/Los_Angeles" TimeZone in HiveCompatibilitySuite
When run the udf_unix_timestamp of
org.apache.spark.sql.hive.execution.HiveCompatibilitySuite testcase
with not "America/Los_Angeles" TimeZone throws error.
[https://issues.apache.org/jira/browse/SPARK-3065]
add locale setting on beforeAll and afterAll method to fix the bug of
HiveCompatibilitySuite testcase
Author: luogankun <[email protected]>
Closes #1968 from luogankun/SPARK-3065 and squashes the following commits:
c167832 [luogankun] [SPARK-3065][SQL] Add Locale setting to
HiveCompatibilitySuite
0a25e3a [luogankun] [SPARK-3065][SQL] Add Locale setting to
HiveCompatibilitySuite
(cherry picked from commit 65253502b913f390b26b9b631380b2c6cf1ccdf7)
Signed-off-by: Michael Armbrust <[email protected]>
commit 9a62cf3655dcab49b5c0f94ad094603eaf288251
Author: Michael Armbrust <[email protected]>
Date: 2014-08-27T22:14:08Z
[SPARK-3235][SQL] Ensure in-memory tables don't always broadcast.
Author: Michael Armbrust <[email protected]>
Closes #2147 from marmbrus/inMemDefaultSize and squashes the following
commits:
5390360 [Michael Armbrust] Merge remote-tracking branch 'origin/master'
into inMemDefaultSize
14204d3 [Michael Armbrust] Set the context before creating
SparkLogicalPlans.
8da4414 [Michael Armbrust] Make sure we throw errors when leaf nodes fail
to provide statistcs
18ce029 [Michael Armbrust] Ensure in-memory tables don't always broadcast.
(cherry picked from commit 7d2a7a91f263bb9fbf24dc4dbffde8fe5e2c7442)
Signed-off-by: Michael Armbrust <[email protected]>
commit 0c03fb621e5b080f24863cfc17032bd828b65b99
Author: Patrick Wendell <[email protected]>
Date: 2014-08-27T22:48:00Z
Revert "[maven-release-plugin] prepare for next development iteration"
This reverts commit 9af3fb7385d1f9f221962f1d2d725ff79bd82033.
commit 0b17c7d4f2176f0c0e8aaab95e034be54467ff30
Author: Patrick Wendell <[email protected]>
Date: 2014-08-27T22:48:13Z
Revert "[maven-release-plugin] prepare release v1.1.0-snapshot2"
This reverts commit e1535ad3c6f7400f2b7915ea91da9c60510557ba.
commit d4cf7a068da099f0f07f04a834d7edf6b743ceb3
Author: Matthew Farrellee <[email protected]>
Date: 2014-08-27T22:50:30Z
Add line continuation for script to work w/ py2.7.5
Error was -
$ SPARK_HOME=$PWD/dist ./dev/create-release/generate-changelist.py
File "./dev/create-release/generate-changelist.py", line 128
if day < SPARK_REPO_CHANGE_DATE1 or
^
SyntaxError: invalid syntax
Author: Matthew Farrellee <[email protected]>
Closes #2139 from mattf/master-fix-generate-changelist.py-0 and squashes
the following commits:
6b3a900 [Matthew Farrellee] Add line continuation for script to work w/
py2.7.5
(cherry picked from commit 64d8ecbbe94c47236ff2d8c94d7401636ba6fca4)
Signed-off-by: Patrick Wendell <[email protected]>
commit 8597e9cf356b0d8e17600a49efc4c4a0356ecb5d
Author: Patrick Wendell <[email protected]>
Date: 2014-08-27T22:55:59Z
BUILD: Updating CHANGES.txt for Spark 1.1
commit 58b0be6a29eab817d350729710345e9f39e4c506
Author: Patrick Wendell <[email protected]>
Date: 2014-08-27T23:28:08Z
[maven-release-plugin] prepare release v1.1.0-rc1
commit 78e3c036eee7113b2ed144eec5061e070b479e56
Author: Patrick Wendell <[email protected]>
Date: 2014-08-27T23:28:27Z
[maven-release-plugin] prepare for next development iteration
commit 54ccd93e621c1bc4afc709a208b609232ab701d1
Author: Andrew Or <[email protected]>
Date: 2014-08-28T06:03:46Z
[HOTFIX] Wait for EOF only for the PySpark shell
In `SparkSubmitDriverBootstrapper`, we wait for the parent process to send
us an `EOF` before finishing the application. This is applicable for the
PySpark shell because we terminate the application the same way. However if we
run a python application, for instance, the JVM actually never exits unless it
receives a manual EOF from the user. This is causing a few tests to timeout.
We only need to do this for the PySpark shell because Spark submit runs as
a python subprocess only in this case. Thus, the normal Spark shell doesn't
need to go through this case even though it is also a REPL.
Thanks davies for reporting this.
Author: Andrew Or <[email protected]>
Closes #2170 from andrewor14/bootstrap-hotfix and squashes the following
commits:
42963f5 [Andrew Or] Do not wait for EOF unless this is the pyspark shell
(cherry picked from commit dafe343499bbc688e266106e4bb897f9e619834e)
Signed-off-by: Patrick Wendell <[email protected]>
commit 233c283e3d946bdcbf418375122c5763559c0119
Author: Michael Armbrust <[email protected]>
Date: 2014-08-28T06:05:34Z
[HOTFIX][SQL] Remove cleaning of UDFs
It is not safe to run the closure cleaner on slaves. #2153 introduced this
which broke all UDF execution on slaves. Will re-add cleaning of UDF closures
in a follow-up PR.
Author: Michael Armbrust <[email protected]>
Closes #2174 from marmbrus/fixUdfs and squashes the following commits:
55406de [Michael Armbrust] [HOTFIX] Remove cleaning of UDFs
(cherry picked from commit 024178c57419f915d26414e1b91ea0019c3650db)
Signed-off-by: Patrick Wendell <[email protected]>
commit ad0fab23d810f0f62e71813edfc3101fcec5e40b
Author: Patrick Wendell <[email protected]>
Date: 2014-08-27T22:40:40Z
HOTFIX: Don't build with YARN support for Mapr3
commit 79e86ef3e1a3ee03a7e3b166a5c7dee11c6d60d7
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T06:45:54Z
[maven-release-plugin] prepare release v1.1.0-rc1
commit a118ea5c59d653f5a3feda21455ba60bc722b3b1
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T06:46:02Z
[maven-release-plugin] prepare for next development iteration
commit c0e3bc17ede69f792cd96556bb3d8acd98b239ce
Author: Cheng Lian <[email protected]>
Date: 2014-08-28T07:08:09Z
[SQL] Fixed 2 comment typos in SQLConf
Author: Cheng Lian <[email protected]>
Closes #2172 from liancheng/sqlconf-typo and squashes the following commits:
115cc71 [Cheng Lian] Fixed 2 comment typos in SQLConf
(cherry picked from commit 68f75dcdfe7e8ab229b73824692c4b3d4c39946c)
Signed-off-by: Michael Armbrust <[email protected]>
commit 2e8ad9976c10a5a7fdefba9b612fc61e48f2b6cc
Author: Michael Armbrust <[email protected]>
Date: 2014-08-28T07:15:23Z
[SPARK-3230][SQL] Fix udfs that return structs
We need to convert the case classes into Rows.
Author: Michael Armbrust <[email protected]>
Closes #2133 from marmbrus/structUdfs and squashes the following commits:
189722f [Michael Armbrust] Merge remote-tracking branch 'origin/master'
into structUdfs
8e29b1c [Michael Armbrust] Use existing function
d8d0b76 [Michael Armbrust] Fix udfs that return structs
(cherry picked from commit 76e3ba4264c4a0bc2c33ae6ac862fc40bc302d83)
Signed-off-by: Michael Armbrust <[email protected]>
commit 71ec0140f7e121bdba3d19e8219e91a5e9d1e320
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T07:15:26Z
Revert "[maven-release-plugin] prepare for next development iteration"
This reverts commit a118ea5c59d653f5a3feda21455ba60bc722b3b1.
commit 56070f12f455bae645cba887a74c72b12f1085f8
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T07:15:30Z
Revert "[maven-release-plugin] prepare release v1.1.0-rc1"
This reverts commit 79e86ef3e1a3ee03a7e3b166a5c7dee11c6d60d7.
commit a9df703081449c41c32cc982781fe1922eaf91ad
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T07:19:03Z
Additional CHANGES.txt
commit da4b94c86c9dd0d624b3040aa4b9449be9f60fc3
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T07:50:32Z
[maven-release-plugin] prepare release v1.1.0-rc1
commit 96926c5a42c5970ed74c50db5bd9c68cacf92207
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T07:50:43Z
[maven-release-plugin] prepare for next development iteration
commit 473b02dfdd8e616717d9d10e4d88e3274e8c8e03
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T08:55:24Z
Revert "[maven-release-plugin] prepare for next development iteration"
This reverts commit 96926c5a42c5970ed74c50db5bd9c68cacf92207.
commit ecdbeeff8d62c6e0cdd4f998f76628e5692eb0c3
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T08:55:31Z
Revert "[maven-release-plugin] prepare release v1.1.0-rc1"
This reverts commit da4b94c86c9dd0d624b3040aa4b9449be9f60fc3.
commit 4186c45f5bc3b72c87d45bdd791983d41968dc34
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T08:55:33Z
Revert "Revert "[maven-release-plugin] prepare release v1.1.0-rc1""
This reverts commit 56070f12f455bae645cba887a74c72b12f1085f8.
commit df6194403a8431048521db1883f9b6feb9ec9983
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T08:55:36Z
Revert "Revert "[maven-release-plugin] prepare for next development
iteration""
This reverts commit 71ec0140f7e121bdba3d19e8219e91a5e9d1e320.
commit d01b3faf896c7e5e26bd1a8d1f9f125097e4dd6f
Author: Patrick Wendell <[email protected]>
Date: 2014-08-28T08:55:41Z
Revert "[maven-release-plugin] prepare for next development iteration"
This reverts commit a118ea5c59d653f5a3feda21455ba60bc722b3b1.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
