Github user budde commented on the issue:
https://github.com/apache/spark/pull/16944
@viirya I've updated the PR to include the initial catalog table checks
you've suggested in the
[```setupCaseSensitiveTable()```](https://github.com/apache/spark/pull/16944/files#diff
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16944
Pinging @viirya and @ericl to take a look at the updates per their feedback
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16744#discussion_r102338119
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala
---
@@ -78,8 +70,9 @@ case class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16744#discussion_r102338189
--- Diff: python/pyspark/streaming/kinesis.py ---
@@ -37,7 +37,8 @@ class KinesisUtils(object):
def createStream(ssc, kinesisAppName, streamName
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
@brkyvz Anyone I can ping to help get this merged? The PR is going on a
month old at this point and I know that lack of STS support is an issue that
several interested parties would like to see get
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
@brkyvz I share your concerns around expanding this API further than
necessary. I think I'm okay with this as long as we're fairly confident the
builder pattern work can be merged in the same Spark
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
@brkyvz I've updated the PR per your feedback. ```BasicAWSCredentials```
will raise a ```java.lang.IllegalArgumentException``` if either keypair value
is null so I elected to wrap
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
Missed updating a test, my mistake. Fixing now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
@brkyvz Any thoughts on moving the dependency version bump to a new commit
and backporting to 2.11 with the pervious versions?
@zswing Any chance you could take a look at this sometime
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16797
@mallman The Parquet schema merging methods take me back to #5214 :)
I haven't been following changes here very closely but I would guess use of
this method was replaced to the previously
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
@brkyvz Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
@brkyvz Would it be possible to backport to 2.1.1 if I reverted to the old
version of the KCL and made the dependency upgrade as a separate PR? We'd still
be adding ```aws-java-sdk-sts
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16797
@cloud-fan:
> Spark does support mixed-case-schema tables, and it has always been. It's
because we write table schema to metastore case-preserving, via table
properties.
Spark pr
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16744
Pinging @brkyvz and @srowen once more for a final look and to get Jenkins
to retest the latest update (not sure if this still requires Jenkins admin
rights).
---
If your project is set up
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r101562475
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -296,6 +296,17 @@ object SQLConf {
.longConf
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r101560890
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -161,23 +161,49 @@ private[hive] class
HiveMetastoreCatalog
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r101605711
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
---
@@ -163,6 +163,10 @@ case class BucketSpec(
* @param
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r101606197
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -186,8 +212,7 @@ private[hive] class HiveMetastoreCatalog
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r101605728
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -296,6 +296,17 @@ object SQLConf {
.longConf
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16944
I've updated the PR based on feedback received. Changes from previous
commit:
- Fixed a couple indent issues
- Clarify some HiveSchemaInferenceSuite comments and general cleanup
- Add
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r101625724
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -296,6 +296,21 @@ object SQLConf {
.longConf
Github user budde closed the pull request at:
https://github.com/apache/spark/pull/16942
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16942
Accidentally did a force-push to my branch for this issue. Looks like I'll
have to open a new PR...
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16942
Tests appear to be failing due to the following error:
```
[info] Exception encountered when attempting to run a suite with class
name
GitHub user budde opened a pull request:
https://github.com/apache/spark/pull/16944
[SPARK-19611][SQL] Introduce configurable table schema inference
*Update: Accidentally broke #16942 via a force push. Opening a replacement
PR.*
Replaces #16797. See the discussion
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16942
@mallman If I did close it then it was by mistake. The "Reopen and comment"
button was disabled with a message about the PR being closed by a force push
when I hovered over it. Afraid
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16944
Re-pinging participants from #16797: @gatorsmile, @viirya, @ericl, @mallman
and @cloud-fan. Sorry for the noise.
---
If your project is set up for it, you can reply to this email and have your
reply
GitHub user budde opened a pull request:
https://github.com/apache/spark/pull/16942
[SPARK-19611][SQL] Introduce configurable table schema inference
Replaces #16797. See the discussion in this PR for more
details/justification for this change.
## Summary of changes
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16942#discussion_r101366307
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSchemaInferenceSuite.scala
---
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16942#discussion_r101366441
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -296,6 +296,17 @@ object SQLConf {
.longConf
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16942#discussion_r101366583
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -296,6 +296,17 @@ object SQLConf {
.longConf
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16797
Thanks for all the feedback on this PR, folks. I'm going to close this
PR/JIRA and open new ones for enabling configurable schema inference as a
fallback. I'll ping each of you who has been active
Github user budde closed the pull request at:
https://github.com/apache/spark/pull/16797
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16942
Pinging participants from #16797: @gatorsmile, @viirya, @ericl, @mallman
and @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16944
Thanks for the thorough feedback.
In the latest update I refactored ```convertToLogicalRelation()``` to
update the schema of the ```catalogTable``` field of the value it returns
whenever
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r103116509
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -226,6 +258,41 @@ private[hive] class HiveMetastoreCatalog
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r103116512
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -161,22 +164,51 @@ private[hive] class
HiveMetastoreCatalog
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r103116519
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSchemaInferenceSuite.scala
---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r103116521
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSchemaInferenceSuite.scala
---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r103116516
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSchemaInferenceSuite.scala
---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r103116517
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSchemaInferenceSuite.scala
---
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107768752
--- Diff:
external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java
---
@@ -0,0 +1,63
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17250
@brkyvz Updated per your feedback, thanks for taking a thorough look. I
also renamed the ```longLivedCredsProvider``` of ```STSCredentials``` to just
```longLivedCreds``` to match the updated naming
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17250
@brkyvz Sure, want me to add it to this PR or open a new one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107971313
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentials.scala
---
@@ -0,0 +1,175 @@
+/*
+ * Licensed
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107975990
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentials.scala
---
@@ -0,0 +1,179 @@
+/*
+ * Licensed
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107977628
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentials.scala
---
@@ -0,0 +1,179 @@
+/*
+ * Licensed
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17250
@brkyvz Awesome, thanks for reviewing this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17250
@brkyvz Updated per your feedback. Most significant change is renaming
```SerializableCredentialsProvider``` to ```SparkAWSCredentials``` (as well as
renaming its subclasses) and refactoring its
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107759010
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -71,7 +75,238 @@ private[kinesis] class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107758905
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -71,7 +75,238 @@ private[kinesis] class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107759287
--- Diff:
external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java
---
@@ -0,0 +1,63
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107759395
--- Diff:
external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisInputDStreamBuilderSuite.scala
---
@@ -0,0 +1,112
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107759339
--- Diff:
external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisInputDStreamBuilderSuite.scala
---
@@ -0,0 +1,112
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107759722
--- Diff:
external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentialsBuilderSuite.scala
---
@@ -0,0 +1,112
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107759746
--- Diff:
external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentialsBuilderSuite.scala
---
@@ -0,0 +1,112
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107760106
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -22,24 +22,28 @@ import
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107759778
--- Diff:
external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentialsBuilderSuite.scala
---
@@ -0,0 +1,112
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107761342
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -71,7 +75,238 @@ private[kinesis] class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107762239
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -22,24 +22,28 @@ import
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107764949
--- Diff:
external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java
---
@@ -0,0 +1,63
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17250
@brkyvz A conference took up a lot of my time last week but I should have
it updated later today
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107025906
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -71,7 +75,256 @@ private[kinesis] class
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17250
@brkyvz Updated the PR to remove ```defaultMessageHandler()``` from
```KinesisUtils``` in favor of keeping this method in
```KinesisInputDStream```. My thought here was that this would be a better
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17250
@brkyvz PR has been updated, apologies for the delay. I've added
```SerializableCredentialsProvider.Builder```, which I'm willing to hear
suggestions for a better name on. I wanted to stay away from
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107318916
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SerializableCredentialsProvider.scala
---
@@ -83,3 +84,146 @@ private
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107318150
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -71,7 +75,256 @@ private[kinesis] class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107318178
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -71,7 +75,256 @@ private[kinesis] class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107318157
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -71,7 +75,256 @@ private[kinesis] class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107318164
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SerializableCredentialsProvider.scala
---
@@ -83,3 +84,146 @@ private
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107318994
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SerializableCredentialsProvider.scala
---
@@ -83,3 +84,146 @@ private
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17250#discussion_r107319004
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SerializableCredentialsProvider.scala
---
@@ -83,3 +84,146 @@ private
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112565900
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -249,6 +252,17 @@ object
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112566462
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -147,6 +152,14 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112343978
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -147,6 +152,14 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112344999
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -249,6 +252,17 @@ object
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112344746
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala
---
@@ -249,6 +252,17 @@ object
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112343115
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -83,7 +83,8 @@ class
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17467
Not a Spark committer, but I've contributed to this component in the past.
I would strongly prefer an approach that avoids adding an additional parameter
to all of the Kinesis classes
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112839963
--- Diff: docs/streaming-kinesis-integration.md ---
@@ -216,3 +216,7 @@ de-aggregate records during consumption.
- If no Kinesis checkpoint info exists
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17467
@yssharma Fair enough. I'll try to get your update reviewed later today
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112817123
--- Diff: docs/streaming-kinesis-integration.md ---
@@ -216,3 +216,7 @@ de-aggregate records during consumption.
- If no Kinesis checkpoint info exists
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112816822
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -295,6 +306,23 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112816810
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -295,6 +306,23 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112816898
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -295,6 +306,23 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112816922
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -295,6 +306,23 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112764344
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -83,7 +86,8 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112766374
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -147,6 +153,17 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112766633
--- Diff:
external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDDSuite.scala
---
@@ -101,6 +101,37 @@ abstract class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112764808
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -147,6 +153,17 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112764788
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -147,6 +153,17 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112764350
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -112,7 +116,8 @@ class
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112765111
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -17,21 +17,24 @@
package
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/17467#discussion_r112765206
--- Diff:
external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala
---
@@ -17,21 +17,24 @@
package
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/16944#discussion_r105270783
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSchemaInferenceSuite.scala
---
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16944
@dongjoon-hyun Hmm, we should be catching and logging this in [this catch
block](https://github.com/apache/spark/blob/f79371ad86d94da14bd1ddb53e99a388017b6892/sql/hive/src/main/scala/org/apache/spark
GitHub user budde opened a pull request:
https://github.com/apache/spark/pull/17229
[SPARK-19611][SQL] Introduce configurable table schema inference
Add a new configuration option that allows Spark SQL to infer a
case-sensitive schema from a Hive Metastore table's data files when
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17229
Backport of #16944 to ```branch-2.1```. Pinging @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user budde commented on the issue:
https://github.com/apache/spark/pull/17229
Made a quick fix to remove some dead code after merging w/the 2.1 version
of ```HiveMetastoreCatalog```
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user budde commented on the issue:
https://github.com/apache/spark/pull/16944
@dongjoon-hyun Are you using Apache Hive itself to run the ```ALTER
TABLE...``` statement? Are you directing the Hive client to use the warehouse
dir created by Spark?
---
If your project is set up
101 - 200 of 302 matches
Mail list logo
