[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17956
Given the existing logic below:
```
if (lowerCaseValue.equals("nan") ||
lowerCaseValue.equals("infinity") ||
lowerCaseValue.equals("-infinity") ||
lowerCaseValue.equals("inf") ||
lowerCaseValue.equals("-inf")) {
value.toFloat
```
It looks we have try all the combinations (lowercase and uppcass) of
`"nan", "infinity", "-infinity", "inf", "-inf"` sofar.
```python
import itertools
items = ["nan", "infinity", "-infinity", "inf", "-inf"]
comb = []
for item in items:
comb.extend(map(''.join, itertools.product(*zip(item.upper(),
item.lower()
print("Set(%s)" % ", ".join(map(lambda c: '"%s"' % c, comb)))
```
This code is to generate the combinations. (I feel convenient for dealing
with strings in Python).
```scala
val s = Set("NAN", "NAn", ... "-inf")
s.foreach { w =>
try {
w.toDouble // or w.toFloat
println(w)
} catch {
case _ => // pass
}
}
```
It looks we have supported the cases below so far:
```
NaN
-Infinity
Infinity
```
Let me update this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17956
Yea, Let's focus on the topic. For the cases below:
```
{"a": NaN}
{"a": Infinity}
{"a": +Infinity}
{"a": -Infinity}
```
They are related with `allowNonNumericNumbers` and related tests are
already being ignored.
Okay. So the point is behaviour change, right? Let me give a shot to remove
the behaviour changes here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17956
BTW, if we want to support the JSON strings like `{"a": "-Infinity"}` as
the FLOAT type, I think we also should support the float data in a string.
Below is an example.
```Scala
def floatRecords: Dataset[String] =
spark.createDataset(spark.sparkContext.parallelize(
"""{"f": "18.00"}""" ::
"""{"f": "18.30"}""" ::
"""{"f": "20.00"}""" :: Nil))(Encoders.STRING)
val jsonDF = spark.read.schema("f float").json(floatRecords)
```
If we want to support parsing float data in a string, we should also
support the other data types. Thus, this is another to-do issue, we should do
an investigation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17956
Based on my previous work experience, I am very conservative to introduce
any behavior change unless we are sure this is the right thing to do. I do not
think this is a high priority PR. I can put it in my to-do list. When I am
free, I can work on it.
To prove the significance of this change, we need to check whether the
other projects/products have the same/similar behavior? We do not want to
introduce anything new or different. This is my concern.
Below is the list we need to check.
```
{"a": "+INF"}
{"a": "INF"}
{"a": "-INF"}
{"a": "NaN"}
{"a": "+NaN"}
{"a": "-NaN"}
{"a": "Infinity"}
{"a": "+Infinity"}
{"a": "-Infinity"}
```
Note, the above is different from the following JSON strings.
```
{"a": NaN}
{"a": Infinity}
{"a": +Infinity}
{"a": -Infinity}
```
In my previous comment, I just want to share something for the people who
want to be a committer in Spark. It is not related to this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 1. It is not good to hold off PRs. If a PR looks good and coherent, I think we could merge. 2. "How many JSON writers write a float "INF" as a string?", if it is your worry, I will take out the all newly added cases here. This is why I am thinking we should add this case via options. 3. My question was to ask taking over this. Please answer to this question. 4. Other information looks not related with the topic in this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17956 Let me share my understanding. First, we do not have a deadline for any non-urgent fixes. This PR is an example. How many JSON writers write a float "INF" as a string? Being a committer needs to make a judgement to see how critical a PR is. My personal judgement is that we do not need to rush to merge it. I even want to wait for understanding how the other platforms handle **such a string-represented "INF" in a float-type column**. I believe we are not the first platform to face such an issue. That is why @cloud-fan and I asked the question above. Second, we have to be careful to avoid introducing bugs when fixing the old issue. Spark is an infrastructure open-source project. We have to be very careful when doing the code reviews and code changes. If we accidentally introduced some supports, it is painful to remove the support and thus we have to keep it maybe forever. @viirya just found it very recently. Thus, please be careful when you do the code review and code changes. If the users really need such a fix, they also can fix it and make a build by themselves. I believe that is why people like open source projects. At the end, I want to share you a slides about bug costs. You might already saw similar figure before. https://cloud.githubusercontent.com/assets/11567269/26012021/00f34d82-3708-11e7-9271-1a97ecef0fff.png";> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 WDYT? I think it'd be faster to get this merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 Hey @gatorsmile, how about picking up the commits here and opening a PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17956 We really need to be careful when making the code changes. If we support such a feature, we need to support all the data types. This is unexpected to users. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 Yea, I am aware of it because we support special floating cases in string and I think we'd expect other doubles could be in string? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17956
This PR also introduces the following behavior changes.
```Scala
def floatRecords: Dataset[String] =
spark.createDataset(spark.sparkContext.parallelize(
"""{"f": "18.00"}""" ::
"""{"f": "18.30"}""" ::
"""{"f": "20.00"}""" :: Nil))(Encoders.STRING)
val jsonDF = spark.read.schema("f float").json(floatRecords)
```
Before this PR, the output is like
```
++
| f|
++
|null|
|null|
|null|
++
```
After this PR, the output is like
```
++
| f|
++
|18.00|
|18.30|
|20.00|
++
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 `allowNonNumericNumbers` option is undocumented and incomplete for now - https://github.com/apache/spark/pull/9724#r44883828 > allowNonNumericNumbers is undocumented for now, since I can't figure out how it works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17956
Before this PR, the option `allowNonNumericNumbers` affects the results of
the following four cases:
```Json
{"a": NaN}
{"a": Infinity}
{"a": +Infinity}
{"a": -Infinity}
```
We should also enable this flag for the following cases too.
```Json
{"a": "+INF"}
{"a": "INF"}
{"a": "-INF"}
{"a": "NaN"}
{"a": "+NaN"}
{"a": "-NaN"}
{"a": "Infinity"}
{"a": "+Infinity"}
{"a": "-Infinity"}
```
In addition, we almost introduced a very serious regression defect, because
we do not have the test case to cover the following scenarios:
```Json
{"a": "4.0"}
```
Thus, we also should include the above test case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17956 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76858/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17956 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17956 **[Test build #76858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76858/testReport)** for PR 17956 at commit [`fa63ff4`](https://github.com/apache/spark/commit/fa63ff4e63bfcb3d780183528663223ab63891a3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17956 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17956 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76857/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17956 **[Test build #76857 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76857/testReport)** for PR 17956 at commit [`d581467`](https://github.com/apache/spark/commit/d5814678e488f6a28c33f5e8a31fde9d6d0b25a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17956 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76854/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17956 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17956 **[Test build #76854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76854/testReport)** for PR 17956 at commit [`e858789`](https://github.com/apache/spark/commit/e85878917d6ff1fe107efc59bb1d403401a2441f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 (Ah, it was about Jackson's `JsonParser.Feature.ALLOW_NON_NUMERIC_NUMBERS` ... I see.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17956
> Do we need to follow allowNonNumericNumbers when parsing """{"a":
"-Infinity"}"""? cc @cloud-fan @viirya
`allowNonNumericNumbers` supports it and Scala supports it too. It seems to
me that we should follow them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 I didn't understand what ^ means. Could you elaborate please? `allowNonNumericNumbers` option here for now is incomplete up to my knowledge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17956
Do we need to follow `allowNonNumericNumbers` when parsing ` """{"a":
"-Infinity"}"""`? cc @cloud-fan @viirya
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17956 uh... That is different... This is String. That is Float. nvm. They are different issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 Yes, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17956 If we do not upgrade `Jackson`, is that possible we can do it by checking the flag `allowNonNumericNumbers` before [doing the conversion](https://github.com/HyukjinKwon/spark/blob/fa63ff4e63bfcb3d780183528663223ab63891a3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala#L129-L131)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 We should upgrade Jackson accordingly to support that option correctly. This looks reverted due to dependency problem - https://github.com/apache/spark/pull/9759#issuecomment-222759764. It sounds a bigger problem to me. (Also, strictly, it looks rather related with parsing itself). We already support the special floating cases and it might be fine just to improve this case. I updated PR title and description already. I will fix JIRA as well (I was hesitated because that was not open by me). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17956 Could we update the JIRA, PR title, and PR description? Also check the JSON flag `allowNonNumericNumbers` when doing the conversion? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17956 **[Test build #76858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76858/testReport)** for PR 17956 at commit [`fa63ff4`](https://github.com/apache/spark/commit/fa63ff4e63bfcb3d780183528663223ab63891a3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17956 **[Test build #76857 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76857/testReport)** for PR 17956 at commit [`d581467`](https://github.com/apache/spark/commit/d5814678e488f6a28c33f5e8a31fde9d6d0b25a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17956 @gatorsmail, I added some more cases I could identify based on http://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#valueOf(java.lang.String) (this looks calling `java.lang.Double.parseDouble` which `toDouble` calls). And also, for sure, I made this falls back to `toDouble` or `toFloat` and throw the same exceptions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17956: [SPARK-18772][SQL] Avoid unnecessary conversion try and ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17956 **[Test build #76854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76854/testReport)** for PR 17956 at commit [`e858789`](https://github.com/apache/spark/commit/e85878917d6ff1fe107efc59bb1d403401a2441f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
