Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/22813
@srowen Your suggestion is very good, but sometimes the maintenance
engineers have limited skills in this area. If they configures the operating
system root directory as `SPARK_WORK_DIR` due
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/22876
@srowen CC
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/22813
@dongjoon-hyun Yes, you can think so. So I want to solve this problem on
the spark platform to reduce the risk of some misoperations of operation and
maintenance engineer. `WorkDirCleanUp
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/22813
As far as I know, when a spark program is submitted to the cluster, a
directory will be created under `SPARK_WORK_DIR`. The directory name consists
of application, timestamp, and five-digit
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/22813
Yes, it happened in our real environment.
The scenario as follows:
Some disk corruption in the production cluster which is normal.
SPARK_LOCAL_DIRS = /data1/bigdata/spark/tmp
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/22813
[SPARK-25818][CORE] WorkDirCleanup should only remove the directory at the
beginning of tâ¦
## What changes were proposed in this pull request?
The cleanup mechanism will clear
Github user ouyangxiaochen closed the pull request at:
https://github.com/apache/spark/pull/0
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/0
[SPARK-25229][SQL] Partition fields are uniformly processed into lowercase
## What changes were proposed in this pull request?
`scala> spark.version
res0: String = 2.
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/21881#discussion_r206377644
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -337,7 +337,11 @@ case class LoadDataCommand
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/21881
@gatorsmile Hi, i am not sure how to build this scene in test case, just
assert if the exception info contains the key message `have no permission to
access the input path
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/21881
[SPARK-24930][SQL] Improve exception information when using LOAD DATA LOCAL
INPATH
## What changes were proposed in this pull request?
## How was this patch tested
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/19380
In fact, there are many scenarios that need to be accurate to milliseconds,
should we try to solve this problem together
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/19380
Since the RDMS keep the milliseconds, we should follow it. This proposal
LGTM. @gatorsmile CC
---
-
To unsubscribe, e
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125651028
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1132,6 +1132,15 @@ private[spark] class
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125575412
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -46,26 +46,47 @@ case class CreateFunctionCommand
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125567012
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -46,26 +46,47 @@ case class CreateFunctionCommand
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125564468
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -46,26 +46,47 @@ case class CreateFunctionCommand
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125563990
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
---
@@ -1316,8 +1316,10 @@ abstract class
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18510#discussion_r125258324
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -1104,10 +1104,10 @@ class SessionCatalog
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125253313
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -590,6 +590,14 @@ class InMemoryCatalog
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r125250502
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -1056,6 +1056,27 @@ class SessionCatalog
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
@gatorsmile @cloud-fan @HyukjinKwon Could you review this PR when you have
sometime? Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
@felixcheung Thank you very much!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
@dongjoon-hyun Thank uï¼ your suspicions about the impl of
`InMemoryCatalog `are correct.
If we add alterFunction test case in ExternalCatalogSuite that we should
impl this in both
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
@SparkQA Please rebuild it , Jerkins encountered some problems, all test
cases run successfully.
@gatorsmile CC
---
If your project is set up for it, you can reply to this email
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r124490206
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -590,6 +590,14 @@ class InMemoryCatalog
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
@SparkQA why failed, retest it please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r124442925
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -590,6 +590,10 @@ class InMemoryCatalog
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r124435476
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
---
@@ -590,6 +590,10 @@ class InMemoryCatalog
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r124429467
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
---
@@ -140,6 +140,16 @@ case class DropFunctionPreEvent
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
OK, I fixed the error in the test case, retest it please, Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
I found only one test case which name is `create temporary function with if
not exists` , but Jenkins throw me an error
`org.scalatest.exceptions.DuplicateTestNameException: Duplicate test
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17681#discussion_r124183913
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala
---
@@ -46,27 +46,53 @@ case class CreateFunctionCommand
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
retest this please,Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
Jenkins, please build and run test.
@gatorsmile review the added test cases,Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
please review it, Thanks @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
OK, I'll work on this PR after your PR #18142 is merged into master.
@gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
Sorry, I have not found the impl of this PR in #18142 .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user ouyangxiaochen reopened a pull request:
https://github.com/apache/spark/pull/17681
[SPARK-20383][SQL] Supporting Create [temporary] Function with the keyword
'IF NOT EXISTS'
## What changes were proposed in this pull request?
support to create [temporary
Github user ouyangxiaochen closed the pull request at:
https://github.com/apache/spark/pull/17681
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
I am sorry about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
Sorry, I have not found the impl of this PR in #18142, so it could not be
closed.
At least the file sqlbase.g4 was not modified.
---
If your project is set up for it, you can reply
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
Since the issue is resolved, i'll close this PR later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
OK. After that, leave me a message, i will work on this. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
The new synax of creating function like this `CREATE (OR REPLACE)?
TEMPORARY? FUNCTION (IF NOT EXISTS)? qualifiedName AS className=STRING (USING
resource (',' resource)*)? `
OK, let's
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
It seems more reasonable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
The synax of DBMS like this` CREATE OR REPLACE FUNCTION `, it means that if
function exists we will replace it, otherwise create one. But i think we'd
better not destroy the original
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
I think it is necessary to support this key word in 'create [temporary]
fucntion ...' clause,
For example:
I hava an application which has three steps as follow :
1. insert
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17681
@gatorsmile CC is it reasonable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/17681
[SPARK-20383][SQL] Supporting Create [temporary] Function with the keyword
'IF NOT EXISTS'
## What changes were proposed in this pull request?
support to create [temporary
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17628
OK, I got it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17628
@srowen Do u mean that my regex can not match all variable name? because
scala's variable name can be consisted with letter|digital|_|$? I serached the
instances of `var foo="s
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17628
I serached the instance of var foo = "string" with the RegEx ` \s*var
[a-z|A-Z]* = "*"` in whole project and there are 31 of them. The usage of them
are almost reason
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17628
OK, I will work on this again, Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17628
I tried to serach the instance of `var foo = "string"` with the RegEx `var
[a-z|A-Z] = "*` in whole project and there are 740 of them. The usage of them
are al
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17628
This is a very heavy task. So far, I found a problem with this when I was
reading SparkSQL source code.
So, whether we need to merge to the master or not? @srowen @viirya
Github user ouyangxiaochen closed the pull request at:
https://github.com/apache/spark/pull/17629
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17628
OK, I'll merge such changes together into this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/17629
[SPARK-20317] [REPL] Removing 'return' keyword in Scala method
## What changes were proposed in this pull request?
Make the code cleaner and remove the redundant code
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/17628
[SPARK-20316][SQL] val and var should strictly follow the Scala syntax
What changes were proposed in this pull request?
val and var should strictly follow the Scala syntax
Github user ouyangxiaochen closed the pull request at:
https://github.com/apache/spark/pull/17270
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17270
Yeah, U are right.
In Hive. the table is created by specifing LOCATION that will be a MANAGED
table.
In Spark2.x. the table is created by specifing LOCATION that will be an
EXTERNAL
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17270
@gatorsmile cc ,is it reasonable? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/17270
As an application developer we want to know hive table's location, we can
use the command 'show create table ...' to get the table's data location when
the hive table is EXTERNAL
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/17270
[SPARK-19929] [SQL] Showing Hive Managed table's LOATION when Using 'show
create table ...'
## What changes were proposed in this pull request?
Show Hive table's LOCATION property
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
Very thoughtful consideration. Thanks for your explanation and suggestion!
@tejasapatil what do you think? @gatorsmile @cloud-fan
---
If your project is set up for it, you can reply
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
Very serious consideration. Thanks for your explanation and suggestion!
what do you think? @gatorsmile @cloud-fan
---
If your project is set up for it, you can reply to this email and have
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
Do you mean that we don't need to do check whether the
targetTable.storage.locationUri is the same with
sourceTable.storage.locationUri or not ? @tejasapatil
---
If your project is set
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
In @tejasapatil's comment, Whether we need to be exactly the same as Hive?
@gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
I think @tejasapatil's suggestion is reasonable, because the location is
specified by users, So the sourceTable.storage.locationUri and
targetTable.storage.locationUri can be same
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
I think there is no need to do this validation, because the location is
specified by users, So the targetTable.storage.lcaotionUri and
sourceTable.storage.locationUri can be same
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16868#discussion_r100712792
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -969,15 +1053,19 @@ class HiveDDLSuite
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
I have run test cases successfully. Please run the test cases again.Thanks
a lot! @SparkQA
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16868#discussion_r100450487
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -926,38 +1006,63 @@ class HiveDDLSuite
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16868#discussion_r100257474
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -833,54 +833,107 @@ class HiveDDLSuite
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I have created a PR at [https://github.com/apache/spark/pull/16868](url),
please review it, Thanks! @gatorsmile @cloud-fan
---
If your project is set up for it, you can reply to this email
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16868
I have created a new PR. Please review it, Thanks! @gatorsmile @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/16868
[SPARK-19115] [SQL] Supporting Create External Table Like Location
What changes were proposed in this pull request?
Support CREATE [EXTERNAL] TABLE LIKE LOCATION... syntax for Hive
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
OK. I'll try it immediately. Thank U very much!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Oh, I See, I miss a step âgit remote add upstream ...â.
But now, I have delete my repository in my profile. So this PR canât know
which repository should be associated. So, do u
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Here's how I create a PR:
1.fork the master of Apache;
2.create a new branch in my master branch
3.select my new branch menu and create a new PR.
4.edit my new branch code
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
My master branch with the master of Apache is not synchronized, and then I
did the pull operation, my master branch still not synchronized, and finally I
removed my remote repository
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Should I delete my local master repository firstlyï¼and fork a new one
again? @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I met some troubles when I resolving the conflict, So can u give me some
guidances? Thanks a lot! @cloud-fan
---
If your project is set up for it, you can reply to this email and have
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r99984076
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -812,150 +812,234 @@ class HiveDDLSuite
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
ping @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Happy Chinese New Year ! @gatorsmile
The Spring Festival holiday just ended, and I return to work today, what
work do I need to do?
---
If your project is set up for it, you can reply
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I have fixed the error of test cases and they run successfully. So,please
run the test cases again.Thanks a lot! @SparkQA
---
If your project is set up for it, you can reply to this email
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r97454893
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1140,14 +1140,18 @@ class SparkSqlAstBuilder(conf
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
I am sorry that I did't grasp the key points of your question. In Hive, if
there are data files under the specified path while creating an external table,
then Hive will identify the files
Github user ouyangxiaochen commented on a diff in the pull request:
https://github.com/apache/spark/pull/16638#discussion_r96993195
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -58,6 +58,7 @@ import org.apache.spark.util.Utils
Github user ouyangxiaochen commented on the issue:
https://github.com/apache/spark/pull/16638
Here is the differences between Hive and Spark2.x as follow:
1.Hive
create table test(id int); --> MANAGED_TABLE
create table test(id int) location '/warehouse/t
GitHub user ouyangxiaochen opened a pull request:
https://github.com/apache/spark/pull/16638
spark-19115
## What changes were proposed in this pull request?
sparksql supports the command : create external table if not exists gen_tbl
like src_tbl location '/warehouse/gen_tbl
93 matches
Mail list logo
