Github user skambha commented on the issue:
https://github.com/apache/spark/pull/23206
Today in Spark, the extension points API `injectOptimizerRule` method
allows the rules to be injected at the end in
`extendedOperatorOptimizationRules` and this becomes 2 batches separated
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/23206
@maropu, Thanks for your question. Yes. Thats correct.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/23206
[SPARK-26249][SQL] Add ability to inject a rule in order and to add a batch
via the Spark Extension Points API
## What changes were proposed in this pull request?
Add two new APIs
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r208089990
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
---
@@ -169,25 +181,50 @@ package object expressions
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
Thanks for the review. I have addressed your comments and pushed the
changes.
@cloud-fan, Please take a look
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r208079529
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -201,7 +204,7 @@ case class Alias(child
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r208078928
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -794,19 +795,37 @@ case class
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r208078754
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -201,7 +204,7 @@ case class Alias(child
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r208059884
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
---
@@ -169,25 +181,50 @@ package object expressions
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207726267
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
---
@@ -169,25 +181,50 @@ package object expressions
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
I have addressed the review comments in this commit
[here](https://github.com/apache/spark/pull/17185/commits/065687f3b987e254f41279d45f0cced6e42e)
@cloud-fan, please take a look
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207718090
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala
---
@@ -169,25 +181,50 @@ package object expressions
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207717569
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -316,8 +345,8 @@ case class UnresolvedRegex
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207717536
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
---
@@ -536,12 +536,13 @@ abstract class
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
The last push has changes that adds the map lookup for the 3 part name. It
implements a solution to address the lookup.. although I think there are more
ways we can go about it and could possibly
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
I rebased and ran the catalyst unit test suite only as a sanity test.
fwiw, I had run the sql and hive and catalyst suites earlier prior to this last
rebase
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207640501
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -262,17 +262,47 @@ abstract class Star extends
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207635526
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -262,17 +262,47 @@ abstract class Star extends
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
Looks like there is another change that has gone in that this PR conflicts
with so the build fails. :(
I will rebase again and push
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207624865
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -121,14 +129,14 @@ abstract class Attribute
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207624726
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -71,19 +71,27 @@ trait NamedExpression
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
Thanks @cloud-fan for the review.
I am working on implementing an idea to get optimized lookup with 3part
name
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
I rebased and found out that the resolution code in Logical plan has
changed and it uses map lookup to do the matching. I have some ideas on how to
incorporate the 3 part name with the map lookup
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r207027392
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala
---
@@ -121,14 +129,14 @@ abstract class Attribute
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r206988504
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
---
@@ -120,22 +120,54 @@ abstract class LogicalPlan
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r206985225
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
---
@@ -120,22 +120,54 @@ abstract class LogicalPlan
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r206978220
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -654,16 +654,19 @@ class SessionCatalog
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r206975896
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -262,17 +262,47 @@ abstract class Star extends
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
@gatorsmile , @cloud-fan, just a quick comment, I have been working on
this and will respond soon.
---
-
To unsubscribe, e
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r165230391
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
---
@@ -88,12 +88,12 @@ case class UnresolvedAttribute
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
I have rebased and pushed the changes. I ran the unit tests ( sql,
catalyst and hive).
Earlier, I was having issues running the hive test suite locally but that
is resolved with the fix from
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
sure. Let me look into it. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19921
Thanks @cloud-fan, @gatorsmile
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19902
Opened a new PR to add the getDouble method.
https://github.com/apache/spark/pull/19921
---
-
To unsubscribe, e-mail: reviews
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/19921
[SPARK-22452][SQL] Add getDouble to DataSourceV2Options
- Implemented getDouble method in DataSourceV2Options
- Add unit test
You can merge this pull request into a Git repository by running
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19902
cool! I will add that and submit a new PR.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19902
great! Thanks @gatorsmile , @cloud-fan.
Yes. I would be happy to open a new PR to add the rest of them.
If I compare with the getXXX types in SparkConf, it looks like the only
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19902
great! Thanks @gatorsmile for your comments.
I have updated with code comments, please take a look. Thanks
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/19902#discussion_r155394195
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceV2Options.java
---
@@ -49,4 +49,23 @@ public DataSourceV2Options(Map<Str
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/19902#discussion_r155389138
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceV2Options.java
---
@@ -49,4 +49,22 @@ public DataSourceV2Options(Map<Str
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/19902#discussion_r155388109
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceV2Options.java
---
@@ -49,4 +49,41 @@ public DataSourceV2Options(Map<Str
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/19902#discussion_r155387973
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/DataSourceV2Options.java
---
@@ -49,4 +49,22 @@ public DataSourceV2Options(Map<Str
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/19902
[SPARK-22452][SQL]Add getInt, getLong, getBoolean to DataSourceV2Options
- Implemented methods getInt, getLong, getBoolean for DataSourceV2Options
- Added new unit tests to exercise
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19747
great! Thank you @gatorsmile, @hvanhovell, @wzhfy
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19747
Thanks @gatorsmile.
I have addressed your comments in the latest commit. Please take a look.
Thanks.
---
-
To unsubscribe
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/19747#discussion_r153377691
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -895,6 +898,19 @@ private[hive] object HiveClientImpl
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19747
Thanks @gatorsmile for your comments.
I have incorporated them in the latest commit:
https://github.com/apache/spark/pull/19747/commits/a1c8a6d308b62f3439f07dbf3257b51855cb09d8
Please
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19747
I have taken care of adding the check in the new
HiveClientImpl.alterTableDataSchema as well and have added some new tests
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/19747#discussion_r151689272
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
---
@@ -40,6 +40,22 @@ class SQLQuerySuite extends QueryTest
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/19747#discussion_r151331207
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -507,6 +508,7 @@ private[hive] class HiveClientImpl
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19747
I synced up and noticed there are some recent changes that have gone in
that changes the alter table schema codepath in the HiveExternalCatalog. I'll
take a look and see what changes might
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/19747
Thanks @wzhfy for your comments. I have addressed them in the latest
commit.
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/19747
[Spark-22431][SQL] Ensure that the datatype in the schema for the
table/view metadata is parseable by Spark before persisting it
## What changes were proposed in this pull request?
* JIRA
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17185#discussion_r106253035
--- Diff:
sql/core/src/test/resources/sql-tests/inputs/columnresolution-views.sql ---
@@ -13,10 +13,8 @@ DROP VIEW view1;
-- Test scenario with Global
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17185
cc @gatorsmile, @cloud-fan I'd really appreciate your review and
comments. Thanks much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/17185
[SPARK-19602][SQL] Support column resolution of fully qualified column name
( 3 part name)
## What changes were proposed in this pull request?
The design details is attached to the JIRA issue
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17067
Thanks a lot Xiao.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17067
- Changes to the SQLQueryTestSuite framework to mask the exprId so I can
add the -ve cases as well using this framework.
- Added -ve test cases to the SQLQueryTestSuite framework and so removed
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103756083
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -52,6 +52,19 @@ abstract class SQLViewSuite extends QueryTest
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103379909
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ColumnResolutionSuite.scala
---
@@ -0,0 +1,173 @@
+/*
+ * Licensed
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103378392
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -52,6 +52,19 @@ abstract class SQLViewSuite extends QueryTest
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17067
Thanks much Xiao for the review and comments.
I have made the following changes:
- Separated out the -ve cases from the +ve cases.
- Moved positive tests and also the cases
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/17067
[SPARK-19602][SQL][TESTS] Add tests for qualified column names
## What changes were proposed in this pull request?
- Add tests covering different scenarios with qualified column names
without
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/16919
Thanks Xiao!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/16919
[SPARK-19585][DOC][SQL] Fix the cacheTable and uncacheTable api call in the
doc
## What changes were proposed in this pull request?
https://spark.apache.org/docs/latest/sql-programming
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/15649
Thanks @cloud-fan, @gatorsmile, @viirya
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/15649#discussion_r85377599
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -65,6 +65,20 @@ class SQLQuerySuite extends QueryTest
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/15649
[SPARK-18121][SQL] Unable to query global temp views when hive support is
enabled
## What changes were proposed in this pull request?
Issue:
Querying on a global temp view throws
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/15168#discussion_r79746339
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -341,6 +342,25 @@ class SQLQuerySuite extends QueryTest
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/15168#discussion_r79716333
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -341,6 +342,25 @@ class SQLQuerySuite extends QueryTest
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/15168#discussion_r79710335
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -341,6 +342,25 @@ class SQLQuerySuite extends QueryTest
Github user skambha closed the pull request at:
https://github.com/apache/spark/pull/13822
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/13822
@cloud-fan, PR 15054 was merged this morning and that has resolved the
issue the PR was trying to address. That said, the changes that went in there
will throw an error that the table does
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/13822#discussion_r79265065
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -660,6 +662,10 @@ case class ShowPartitionsCommand
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/13822#discussion_r79265007
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -595,6 +595,19 @@ class HiveDDLSuite
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/13822#discussion_r79264648
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -595,6 +595,19 @@ class HiveDDLSuite
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/13822
Thank you so much Andrew for reviewing. I appreciate it.
I have taken care of your comments and also rebased. Please take a look.
Thanks.
My changes were doing two things:
1
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/14897#discussion_r77879554
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -451,7 +464,7 @@ class SessionCatalog
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/13822
gentle ping @andrewor14
I have rebased it to the master. When you get a chance, can you please
review. Thanks.
---
If your project is set up for it, you can reply to this email
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/13822
[SPARK-16115][SQL] Change output schema to be partition for SHOW PARTITIONS
command and â¦
## What changes were proposed in this pull request?
Changes include:
1. For the SHOW
Github user skambha closed the pull request at:
https://github.com/apache/spark/pull/11480
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/11775#discussion_r57043148
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
---
@@ -693,23 +693,25 @@ class MetastoreDataSourcesSuite
Github user skambha commented on the pull request:
https://github.com/apache/spark/pull/11775#issuecomment-199904383
Thanks @liancheng for the review, merging the fix and adding me to the
spark contributor group in Apache JIRA.
---
If your project is set up for it, you can reply
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/11775
[SPARK-13774][SQL] - Improve error message for non-existent paths and add
tests
SPARK-13774: IllegalArgumentException: Can not create a Path from an empty
string for incorrect file path
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/11775#discussion_r56549702
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
---
@@ -237,6 +237,8 @@ class SparkHadoopUtil extends Logging
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/11775#discussion_r56689434
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -205,7 +205,16 @@ case class DataSource
Github user skambha commented on the pull request:
https://github.com/apache/spark/pull/11775#issuecomment-198001758
@srowen Thanks for your comments. I have addressed them in the last
commit.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/11775#discussion_r56551299
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -205,7 +205,17 @@ case class DataSource
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/11480
[SPARK-13630][SQL] Adds optimizer rule collapsesorts to collapse adjaâ¦
## What changes were proposed in this pull request?
This patch does the following:
I) Adds a new optimizer
90 matches
Mail list logo
