Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11724
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-218955594
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-218955604
Merging this into master and 2.0, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-218951716
cc @davies can you review this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-218946776
@rxin Do you mind if I ask a quick look again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216724913
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216724912
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216724811
**[Test build #57701 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57701/consoleFull)**
for PR 11724 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216723244
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216723246
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216722888
**[Test build #57698 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57698/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216712652
**[Test build #57701 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57701/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216712032
**[Test build #57698 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57698/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216704194
@rxin I see. Thank you. Let me fix this up and change the description as
well with some rules for `LongType`, `DoubleType` and `DecimalType`.
---
If your project
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216630213
I actually worry that we are inferring things directly as decimals for
floating point numbers, because a lot of formats and tools don't necessarily
handle those well.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216515821
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216515597
**[Test build #57629 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57629/consoleFull)**
for PR 11724 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216513469
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216513466
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216513237
**[Test build #57627 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57627/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216499930
**[Test build #57629 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57629/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216499340
@rxin I added some more commits for unit tests in `CSVInferSchemaSuite`.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216497944
**[Test build #57627 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57627/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216465994
@rxin Sure I will add more explicit description and some tests for this.
Thanks.
---
If your project is set up for it, you can reply to this email and have your
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216465482
@HyukjinKwon unfortunately this is too confusing. Can you precisely
describe the inference rule in the pr description, and create (unit - not end
to end) test cases for
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216095497
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216095498
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216095438
**[Test build #57495 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57495/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-216089483
**[Test build #57495 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57495/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-213242625
@rxin I am willing to close this one if you are not sure of this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-202238512
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-202238511
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-202238353
**[Test build #54295 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54295/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-202220872
The commits I just added include the behaviour below:
Infering Types
- `DecimalType` is tried first. So, `10.1` (scale < precision) and
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-202219568
**[Test build #54295 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54295/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-202219010
@falaki Could you take a look at this please?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r57272004
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -108,14 +109,38 @@ private[csv] object
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56785216
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -108,14 +109,38 @@ private[csv] object
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56385761
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -108,14 +109,38 @@ private[csv] object
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56444849
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -108,14 +109,38 @@ private[csv] object
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197211694
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197211692
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197211296
**[Test build #53292 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53292/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197186758
**[Test build #53292 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53292/consoleFull)**
for PR 11724 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197127277
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197127274
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197127036
**[Test build #53257 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53257/consoleFull)**
for PR 11724 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197123101
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197123098
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197122324
**[Test build #53254 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53254/consoleFull)**
for PR 11724 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197117794
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197117713
**[Test build #53256 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53256/consoleFull)**
for PR 11724 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197117795
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197096044
**[Test build #53257 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53257/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197094981
**[Test build #53256 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53256/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197094083
this test please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197093880
**[Test build #53254 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53254/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56268274
--- Diff: sql/core/src/test/resources/decimal.csv ---
@@ -0,0 +1,4 @@
+decimal
+21602730330601001035858
--- End diff --
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-197089379
Just to make sure that checking precision work fine, the codes below work
correctly.
```scala
import java.math.BigDecimal
import
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56266566
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -86,6 +86,7 @@ private[csv] object
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56266282
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -108,14 +109,38 @@ private[csv] object
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56265404
--- Diff: sql/core/src/test/resources/decimal.csv ---
@@ -0,0 +1,4 @@
+decimal
+21602730330601001035858
--- End diff --
Would be
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56265300
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -108,14 +109,38 @@ private[csv] object
Github user falaki commented on a diff in the pull request:
https://github.com/apache/spark/pull/11724#discussion_r56265250
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -86,6 +86,7 @@ private[csv] object
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196838887
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196838883
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196838408
**[Test build #53194 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53194/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196799243
**[Test build #53194 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53194/consoleFull)**
for PR 11724 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196791029
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196791033
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196790668
**[Test build #53187 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53187/consoleFull)**
for PR 11724 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196751935
**[Test build #53187 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53187/consoleFull)**
for PR 11724 at commit
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196751413
There should be a conflict with https://github.com/apache/spark/pull/11550.
I will resolve the conflict as soon as either this one or that one is
merged.
Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-196749567
cc @rxin @falaki
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/11724
[SPARK-13866][SQL] Handle decimal type in CSV inference at CSV data source.
## What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-13866
75 matches
Mail list logo
