[GitHub] spark issue #16083: [SPARK-18097][SQL] Add exception catch to handle corrupt...
Github user jayadevanmurali commented on the issue: https://github.com/apache/spark/pull/16083 Hi @HyukjinKwon , I tried to corrupt the metadata by editing files under spark/warehouse directory. But it is not working. Do you have any idea to corrupt metadata manually. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16635: [SPARK-19059] [SQL] Unable to retrieve data from parquet...
Github user jayadevanmurali commented on the issue: https://github.com/apache/spark/pull/16635 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16635: [SPARK-19059] [SQL] Unable to retrieve data from parquet...
Github user jayadevanmurali commented on the issue: https://github.com/apache/spark/pull/16635 @cloud-fan Incorporated code review comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16635: [SPARK-19059] [SQL] Unable to retrieve data from ...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/16635#discussion_r96790387
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2513,4 +2513,18 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
}
+
+ test(
+"SPARK-19059: Unable to retrieve data from parquet table whose name
startswith underscore") {
+sql("CREATE TABLE `_tbl`(i INT) USING parquet")
+sql("INSERT INTO `_tbl` VALUES (1), (2), (3)")
+checkAnswer( sql("SELECT * FROM `_tbl`"), Row(1) :: Row(2) :: Row(3)
:: Nil)
--- End diff --
It should be fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16635: [SPARK-19059] [SQL] Unable to retrieve data from ...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/16635#discussion_r96790264
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2513,4 +2513,18 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
}
+
+ test(
+"SPARK-19059: Unable to retrieve data from parquet table whose name
startswith underscore") {
+sql("CREATE TABLE `_tbl`(i INT) USING parquet")
--- End diff --
I will do this change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16635: [SPARK-19059] [SQL] Unable to retrieve data from ...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/16635#discussion_r96790238
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -2513,4 +2513,18 @@ class SQLQuerySuite extends QueryTest with
SharedSQLContext {
}
}
}
+
+ test(
+"SPARK-19059: Unable to retrieve data from parquet table whose name
startswith underscore") {
--- End diff --
Yes, it is not parquet only. I think SPARK-19059: read file based table
whose name starts with underscore is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16635: [SPARK-19059] [SQL] Unable to retrieve data from parquet...
Github user jayadevanmurali commented on the issue: https://github.com/apache/spark/pull/16635 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16635: [SPARK-19059] [SQL] Unable to retrieve data from parquet...
Github user jayadevanmurali commented on the issue: https://github.com/apache/spark/pull/16635 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16635: [SPARK-19059] [SQL] Unable to retrieve data from ...
GitHub user jayadevanmurali opened a pull request: https://github.com/apache/spark/pull/16635 [SPARK-19059] [SQL] Unable to retrieve data from parquet table whose name startswith underscore ## What changes were proposed in this pull request? The initial shouldFilterOut() method invocation filter the root path name(table name in the intial call) and remove if it contains _. I moved the check one level below, so it first list files/directories in the given root path and then apply filter. (Please fill in changes proposed in this fix) ## How was this patch tested? Added new test case for this scenario (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark branch-0.1-SPARK-19059 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16635.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16635 commit 290b86ba76e8f6032824c95079748670c2257db0 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-31T02:28:52Z Merge pull request #1 from apache/master Update from original commit 9a7f87f7bd4b75b5149e826a7a5868f5549132cf Author: jayadevan <jayadeva...@tcs.com> Date: 2016-05-10T04:04:32Z Merge remote-tracking branch 'upstream/master' commit d8ed5862f887ced075bf73f58e1f9c13ff340527 Author: jayadevan <jayadeva...@tcs.com> Date: 2016-10-24T17:32:19Z Merge remote-tracking branch 'upstream/master' commit cffbec2d6c044a7fb568dcb19b28b07158b870ea Author: jayadevan <jayadeva...@tcs.com> Date: 2016-11-26T07:30:36Z Merge remote-tracking branch 'upstream/master' commit e6727500925be983bf7c345f1e3e6ad6756f19cb Author: jayadevan <jayadeva...@tcs.com> Date: 2017-01-11T15:43:52Z Merge remote-tracking branch 'upstream/master' commit f5999b9577b0d2ca27c699dfc7a1662933dd Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2017-01-18T18:25:15Z Added test case to handle SPARK-19059 commit 74e4a1a2740262fcf84e8c0704ddf2df57e614a0 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2017-01-18T18:38:36Z Updated listLeafFiles() handile SPARK-19059 commit dec96d848948d6c5f263b1e5535345d24c1058d3 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2017-01-18T18:52:05Z Update PartitioningAwareFileIndex.scala commit 4334b2ba2a882a94d59459a3d73b5fdb6fda6b80 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2017-01-18T18:58:22Z Update PartitioningAwareFileIndex.scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16083: [SPARK-18097][SQL] Add exception catch to handle ...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/16083#discussion_r90372901
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala ---
@@ -189,14 +189,18 @@ case class DropTableCommand(
if (!catalog.isTemporaryTable(tableName) &&
catalog.tableExists(tableName)) {
// If the command DROP VIEW is to drop a table or DROP TABLE is to
drop a view
// issue an exception.
- catalog.getTableMetadata(tableName).tableType match {
-case CatalogTableType.VIEW if !isView =>
- throw new AnalysisException(
-"Cannot drop a view with DROP TABLE. Please use DROP VIEW
instead")
-case o if o != CatalogTableType.VIEW && isView =>
- throw new AnalysisException(
-s"Cannot drop a table with DROP VIEW. Please use DROP TABLE
instead")
-case _ =>
+ try {
+catalog.getTableMetadata(tableName).tableType match {
+ case CatalogTableType.VIEW if !isView =>
+throw new AnalysisException(
+ "Cannot drop a view with DROP TABLE. Please use DROP VIEW
instead")
+ case o if o != CatalogTableType.VIEW && isView =>
+throw new AnalysisException(
+ s"Cannot drop a table with DROP VIEW. Please use DROP TABLE
instead")
+ case _ =>
+}
+ } catch {
+ case e: QueryExecutionException => log.warn(e.toString, e)
--- End diff --
Hi @gatorsmile, I tested this change against all existing test cases. Also
@davies mentioned in the jira itself, we cannot purposefully corrupt the meta
data. So I tested and verified, this change will not break any existing test
cases. And we are expecting QueryExecutionException when the meta data is get
corrupted in some way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16083: [SPARK-18097][SQL] Add exception catch to handle ...
GitHub user jayadevanmurali opened a pull request: https://github.com/apache/spark/pull/16083 [SPARK-18097][SQL] Add exception catch to handle corrupted metadata scenario while dropping the table ## What changes were proposed in this pull request? Add exception catch to handle corrupted meta data scenario while dropping the table ## How was this patch tested? Unit tests Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark branch-0.1-SPARK-18097 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16083.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16083 commit 290b86ba76e8f6032824c95079748670c2257db0 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-31T02:28:52Z Merge pull request #1 from apache/master Update from original commit 9a7f87f7bd4b75b5149e826a7a5868f5549132cf Author: jayadevan <jayadeva...@tcs.com> Date: 2016-05-10T04:04:32Z Merge remote-tracking branch 'upstream/master' commit d8ed5862f887ced075bf73f58e1f9c13ff340527 Author: jayadevan <jayadeva...@tcs.com> Date: 2016-10-24T17:32:19Z Merge remote-tracking branch 'upstream/master' commit cffbec2d6c044a7fb568dcb19b28b07158b870ea Author: jayadevan <jayadeva...@tcs.com> Date: 2016-11-26T07:30:36Z Merge remote-tracking branch 'upstream/master' commit 14bc0b9e93cc755dce3777ac40a51de0ee3dbc5f Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-11-30T18:30:16Z updated ddl.scala Handling the drop table command for meta data corrupted scenario. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15613: merge to master
Github user jayadevanmurali closed the pull request at: https://github.com/apache/spark/pull/15613 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #15613: merge to master
GitHub user jayadevanmurali opened a pull request: https://github.com/apache/spark/pull/15613 merge to master ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/15613.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #15613 commit 290b86ba76e8f6032824c95079748670c2257db0 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-31T02:28:52Z Merge pull request #1 from apache/master Update from original commit 9a7f87f7bd4b75b5149e826a7a5868f5549132cf Author: jayadevan <jayadeva...@tcs.com> Date: 2016-05-10T04:04:32Z Merge remote-tracking branch 'upstream/master' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12941][SQL][MASTER] Spark-SQL JDBC Orac...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/11306#discussion_r54210286
--- Diff:
docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.math.BigDecimal
+import java.sql.{Connection, Date, Timestamp}
+import java.util.Properties
+
+import org.apache.spark.sql.test.SharedSQLContext
+import org.apache.spark.sql.types._
+import org.apache.spark.tags.DockerTest
+
+/**
+ * This patch was tested using the Oracle docker. Created this integration
suite for the same.
+ * The ojdbc6-11.2.0.2.0.jar was to be downloaded from the maven
repository. Since there was
+ * no jdbc jar available in the maven repository, the jar was downloaded
from oracle site
+ * manually and installed in the local; thus tested. So, for SparkQA test
case run, the
+ * ojdbc jar might be manually placed in the local maven
repository(com/oracle/ojdbc6/11.2.0.2.0)
+ * while Spark QA test run.
+ *
+ * The following would be the steps to test this
+ * 1. Pull oracle 11g image - docker pull wnameless/oracle-xe-11g
+ * 2. Start docker - sudo service docker start
+ * 3. Download oracle 11g driver jar and put it in maven local repo:
+ *(com/oracle/ojdbc6/11.2.0.2.0/ojdbc6-11.2.0.2.0.jar)
+ * 4. The timeout and interval parameter to be increased from 60,1 to a
high value for oracle test
+ *in DockerJDBCIntegrationSuite.scala (Locally tested with 200,200 and
executed successfully).
+ * 5. Run spark test - ./build/sbt "test-only
org.apache.spark.sql.jdbc.OracleIntegrationSuite"
+ *
+ * All tests in this suite are ignored because of the dependency with the
oracle jar from maven
+ * repository.
+ */
+@DockerTest
+class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with
SharedSQLContext {
--- End diff --
@yhuai the beforeall method will not execute because there is no valid test
inside this class(already put ignore on top of test).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12941][SQL][MASTER] Spark-SQL JDBC Orac...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11306#issuecomment-188159399 @SparkQA Seems to be the build failed due kmeans error. I think this is not related to this PR. Could you please check --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-182760091 @hvanhovell Thanks for the quick fix, I think we can retest this PR now. What you think ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-182884877 @hvanhovell I think, you may comment "ok to test" for triggering the build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-182942391 Ok. I think something missing in our test request, lot test started now which all requested after our PR --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-182964460 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-182452861 @hvanhovell Why this much delay for test ? Anything required from my side ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-182461950 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/11051#discussion_r52527852
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -1291,4 +1291,16 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
Seq(1 -> "a").toDF("i", "j").filter($"i".cast(StringType) === "1"),
Row(1, "a"))
}
+
+ test("SPARK-12982: Add table name validation in temp table
registration") {
+val df = Seq("foo", "bar").map(Tuple1.apply).toDF("col")
+// invalid table name test as below
+intercept[AnalysisException](df.registerTempTable("t~"))
+// valid table name test as below
+df.registerTempTable("table1")
+// another invalid table name test as below
+intercept[AnalysisException](df.registerTempTable("#$@sum"))
+// another invalid table name test as below
+intercept[AnalysisException](df.registerTempTable("table!#"))
--- End diff --
Ok thanks for the info. Let me know if you need any.help from.me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/11051#discussion_r52526270
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -1291,4 +1291,16 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
Seq(1 -> "a").toDF("i", "j").filter($"i".cast(StringType) === "1"),
Row(1, "a"))
}
+
+ test("SPARK-12982: Add table name validation in temp table
registration") {
+val df = Seq("foo", "bar").map(Tuple1.apply).toDF("col")
+// invalid table name test as below
+intercept[AnalysisException](df.registerTempTable("t~"))
+// valid table name test as below
+df.registerTempTable("table1")
+// another invalid table name test as below
+intercept[AnalysisException](df.registerTempTable("#$@sum"))
+// another invalid table name test as below
+intercept[AnalysisException](df.registerTempTable("table!#"))
--- End diff --
Is this a new bug ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-181957359 @SparkQA Could you please retest the changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-181266259 Thanks @hvanhovell for this info --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-181260259 @SparkQA Could you please do a test again @hvanhovell @davies Incorporated review comments in test case. Could you please review the same --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-181269499 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-181265819 @hvanhovell I will make that change now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-181701118 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10983#issuecomment-179886072 Created new pr with latest code for this issue --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali closed the pull request at: https://github.com/apache/spark/pull/10983 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/11051#discussion_r51973903
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -1291,4 +1291,14 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
Seq(1 -> "a").toDF("i", "j").filter($"i".cast(StringType) === "1"),
Row(1, "a"))
}
+
+ test("SPARK-12982: Add table name validation in temp table
registration") {
--- End diff --
@hvanhovell @davies
Shall I include the changes for the above review comments now ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/11051#issuecomment-179964614 @SparkQA Could you please test again --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10983#issuecomment-179270485 @hvanhovell I have created a new PR for this issue with latest code. https://github.com/apache/spark/pull/11051. Could you please review the same. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
Github user jayadevanmurali closed the pull request at: https://github.com/apache/spark/pull/11049 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
GitHub user jayadevanmurali opened a pull request: https://github.com/apache/spark/pull/11049 Merge pull request #1 from apache/master Update from original You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11049.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11049 commit 290b86ba76e8f6032824c95079748670c2257db0 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-31T02:28:52Z Merge pull request #1 from apache/master Update from original --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
GitHub user jayadevanmurali opened a pull request: https://github.com/apache/spark/pull/11051 [SPARK-12982][SQL] Add table name validation in temp table registration Add the table name validation at the temp table creation You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark branch-0.2-SPARK-12982 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11051.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11051 commit 290b86ba76e8f6032824c95079748670c2257db0 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-31T02:28:52Z Merge pull request #1 from apache/master Update from original commit 7c7b503f88e0c74e324ef4b2da99f70d2e677a42 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-02-03T14:31:42Z Update SQLContext.scala Add table name validation in temp table registration commit 3407fd785ae1c6c52607c96fb084617f2acd3e46 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-02-03T14:33:46Z Update DataFrameSuite.scala Added test case for validating table name --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
Github user jayadevanmurali closed the pull request at: https://github.com/apache/spark/pull/11049 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Merge pull request #1 from apache/master
GitHub user jayadevanmurali reopened a pull request: https://github.com/apache/spark/pull/11049 Merge pull request #1 from apache/master Update from original You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11049.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11049 commit 290b86ba76e8f6032824c95079748670c2257db0 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-31T02:28:52Z Merge pull request #1 from apache/master Update from original --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/11051#discussion_r51762232
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -1291,4 +1291,14 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
Seq(1 -> "a").toDF("i", "j").filter($"i".cast(StringType) === "1"),
Row(1, "a"))
}
+
+ test("SPARK-12982: Add table name validation in temp table
registration") {
+val rows = List(Row("foo"), Row("bar"))
+val schema = StructType(Seq(StructField("col", StringType)))
+val rdd = sparkContext.parallelize(rows)
+val df = sqlContext.createDataFrame(rdd, schema)
+assert(intercept[AnalysisException](df.registerTempTable("t~"))
+ .getMessage.contains("NoViableAltException"))
--- End diff --
@hvanhovell So we can simply intercept analysis exception instead of
asserting NoViableAltException. Am I right ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/11051#discussion_r51766868
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -1291,4 +1291,14 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
Seq(1 -> "a").toDF("i", "j").filter($"i".cast(StringType) === "1"),
Row(1, "a"))
}
+
+ test("SPARK-12982: Add table name validation in temp table
registration") {
+val rows = List(Row("foo"), Row("bar"))
--- End diff --
@hvanhovell OK I will do the change
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/11051#discussion_r51766777
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
---
@@ -1291,4 +1291,14 @@ class DataFrameSuite extends QueryTest with
SharedSQLContext {
Seq(1 -> "a").toDF("i", "j").filter($"i".cast(StringType) === "1"),
Row(1, "a"))
}
+
+ test("SPARK-12982: Add table name validation in temp table
registration") {
--- End diff --
yes @hvanhovell
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10983#issuecomment-177969053 @hvanhovell, Yea I tried to merge my branch (https://github.com/jayadevanmurali/spark/tree/branch-0.1-SPARK-SPARK-12982) with apache spark master. But I cant see any option to merge by "create pull request". That's why I suggest new PR option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10983#issuecomment-177942271 @hvanhovell the current pull request is based on an outdated code. So I would like to create a new pull request on the latest code. Hope that is fine --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10983#issuecomment-177447251 Thanks @hvanhovell, Working on test case --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10983#issuecomment-177540750 @hvanhovell yes sure, no problem --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request:
https://github.com/apache/spark/pull/10983#issuecomment-177109715
@hvanhovell
I was able to replicate this in spark 2.0.0.
Steps
ayadevan@Satellite-L640:~/spark$ ./bin/spark-shell
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.0.0-SNAPSHOT
/_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java
1.7.0_80)
Type in expressions to have them evaluated.
Type :help for more information.
16/01/30 14:19:21 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
16/01/30 14:19:22 WARN Utils: Your hostname, Satellite-L640 resolves to a
loopback address: 127.0.1.1; using 100.86.225.72 instead (on interface ppp0)
16/01/30 14:19:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
Spark context available as sc (master = local[*], app id =
local-1454143767817).
SQL context available as sqlContext.
scala> import org.apache.spark.sql.types.{StringType, StructField,
StructType}
import org.apache.spark.sql.types.{StringType, StructField, StructType}
scala> import org.apache.spark.sql.{DataFrame, Row, SQLContext}
import org.apache.spark.sql.{DataFrame, Row, SQLContext}
scala> import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.{SparkContext, SparkConf}
scala> val rows = List(Row("foo"), Row("bar"));
rows: List[org.apache.spark.sql.Row] = List([foo], [bar])
scala> val schema = StructType(Seq(StructField("col", StringType)));
schema: org.apache.spark.sql.types.StructType =
StructType(StructField(col,StringType,true))
scala> val rdd = sc.parallelize(rows);
rdd: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] =
ParallelCollectionRDD[0] at parallelize at :32
scala> val df = sqlContext.createDataFrame(rdd, schema)
df: org.apache.spark.sql.DataFrame = [col: string]
scala> df.registerTempTable("t~")
scala> df.sqlContext.dropTempTable("t~")
java.lang.RuntimeException: [1.2] failure: ``.'' expected but `~' found
t~
^
at scala.sys.package$.error(package.scala:27)
at
org.apache.spark.sql.catalyst.SqlParser$.parseTableIdentifier(SqlParser.scala:58)
at org.apache.spark.sql.SQLContext.table(SQLContext.scala:836)
at org.apache.spark.sql.SQLContext.dropTempTable(SQLContext.scala:763)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:39)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:44)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:46)
at $iwC$$iwC$$iwC$$iwC$$iwC.(:48)
at $iwC$$iwC$$iwC$$iwC.(:50)
at $iwC$$iwC$$iwC.(:52)
at $iwC$$iwC.(:54)
at $iwC.(:56)
at (:58)
at .(:62)
at .()
at .(:7)
at .()
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)
at
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)
at
org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at
org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on a diff in the pull request:
https://github.com/apache/spark/pull/10983#discussion_r51342667
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -747,7 +747,7 @@ class SQLContext private[sql](
* only during the lifetime of this instance of SQLContext.
*/
private[sql] def registerDataFrameAsTable(df: DataFrame, tableName:
String): Unit = {
-catalog.registerTable(TableIdentifier(tableName), df.logicalPlan)
+catalog.registerTable(SqlParser.parseTableIdentifier(tableName),
df.logicalPlan)
--- End diff --
Ok I can see the variable definition at line 211 of SqlContext.scala
@transient
protected[sql] val sqlParser = new SparkSQLParser(getSQLDialect().parse(_))
But this varable is not used anywhare.All methods use
Sqlarser.parseTableIdentifier() for example @Experimental
def createExternalTable(
tableName: String,
source: String,
options: Map[String, String]): DataFrame = {
**val tableIdent = SqlParser.parseTableIdentifier(tableName)**
val cmd =
CreateTableUsing(
tableIdent,
userSpecifiedSchema = None,
source,
temporary = false,
options,
allowExisting = false,
managedIfNoPath = false)
executePlan(cmd).toRdd
table(tableIdent)
}
Correct me if am wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
Github user jayadevanmurali commented on the pull request:
https://github.com/apache/spark/pull/10983#issuecomment-177392588
Thanks @hvanhovell , Got your point. I updated my code and repeat the
steps. I was able to replicate this. Please check the steps
jayadevan@Satellite-L640:~/spark$ ./bin/spark-shell
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes
ahead of assembly.
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel).
16/01/31 09:27:57 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
16/01/31 09:27:57 WARN Utils: Your hostname, Satellite-L640 resolves to a
loopback address: 127.0.1.1, but we couldn't find any external IP address!
16/01/31 09:27:57 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to
another address
Spark context available as sc (master = local[*], app id =
local-1454212680541).
SQL context available as sqlContext.
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.0.0-SNAPSHOT
/_/
Using Scala version 2.11.7 (Java HotSpot(TM) 64-Bit Server VM, Java
1.7.0_80)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.spark.sql.types.{StringType, StructField,
StructType}
import org.apache.spark.sql.types.{StringType, StructField, StructType}
scala> import org.apache.spark.sql.{DataFrame, Row, SQLContext}
import org.apache.spark.sql.{DataFrame, Row, SQLContext}
scala> import org.apache.spark.{SparkContext, SparkConf}
import org.apache.spark.{SparkContext, SparkConf}
scala> val rows = List(Row("foo"), Row("bar"));
rows: List[org.apache.spark.sql.Row] = List([foo], [bar])
scala> val schema = StructType(Seq(StructField("col", StringType)));
schema: org.apache.spark.sql.types.StructType =
StructType(StructField(col,StringType,true))
scala> val rdd = sc.parallelize(rows);
rdd: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] =
ParallelCollectionRDD[0] at parallelize at :29
scala> val df = sqlContext.createDataFrame(rdd, schema)
df: org.apache.spark.sql.DataFrame = [col: string]
scala> df.registerTempTable("t~")
scala> df.sqlContext.dropTempTable("t~")
org.apache.spark.sql.AnalysisException: NoViableAltException(327@[209:20: (
DOT id2= identifier )?])
; line 1 pos 1
at
org.apache.spark.sql.catalyst.parser.ParseErrorReporter.throwError(ParseDriver.scala:158)
at
org.apache.spark.sql.catalyst.parser.ParseErrorReporter.throwError(ParseDriver.scala:147)
at
org.apache.spark.sql.catalyst.parser.ParseDriver$.parse(ParseDriver.scala:95)
at
org.apache.spark.sql.catalyst.parser.ParseDriver$.parseTableName(ParseDriver.scala:42)
at
org.apache.spark.sql.catalyst.CatalystQl.parseTableIdentifier(CatalystQl.scala:81)
at org.apache.spark.sql.SQLContext.table(SQLContext.scala:811)
at org.apache.spark.sql.SQLContext.dropTempTable(SQLContext.scala:738)
... 49 elided
So I will close this pull request and raise a new one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12982][SQL] Add table name validation i...
GitHub user jayadevanmurali opened a pull request: https://github.com/apache/spark/pull/10983 [SPARK-12982][SQL] Add table name validation in temp table registration Add the table name validation at the temp table creation You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark branch-0.1-SPARK-SPARK-12982 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10983.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10983 commit ef5d7c72d53d06b517f95201094a21fbb97d0006 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-29T11:55:43Z Update SQLContext.scala Add the table name validation at the temp table creation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11137][Streaming] Make StreamingContext...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10807#issuecomment-173197159 @srowen Thanks for the detailed comment. Can you merge the changes to master branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update StreamingContext.scala [SPARK-11137]
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10807#issuecomment-172595828 Exceptions are handled for each of the operations in the stop method, so that any exceptions does not abort rest of the statements --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11137][Streaming] Make StreamingContext...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10807#issuecomment-172609322 @srowen sure and updated the the title --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update StreamingContext.scala [SPARK-11137]
GitHub user jayadevanmurali opened a pull request: https://github.com/apache/spark/pull/10807 Update StreamingContext.scala [SPARK-11137] Make StreamingContext.stop() exception-safe You can merge this pull request into a Git repository by running: $ git pull https://github.com/jayadevanmurali/spark branch-0.1-SPARK-11137 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10807.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10807 commit 39bcfb58b2e0e63afcdd931f881d6071e246d4c2 Author: jayadevanmurali <jayadeva...@tcs.com> Date: 2016-01-18T16:20:56Z Update StreamingContext.scala [SPARK-11137] Make StreamingContext.stop() exception-safe --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12438][SQL] hiveContext.table() throws ...
Github user jayadevanmurali commented on the pull request: https://github.com/apache/spark/pull/10529#issuecomment-168314723 I think, there would be a reference update in Analyzer.scala as we discussed @thomastechs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
