Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22171
Scientific notation is more efficient on saving the values in CSV. If there
are many zero values of high scale decimal type, this non scientific notation
can cost storage space and loading time.
Github user vinodkc commented on the issue:
https://github.com/apache/spark/pull/22171
@viirya , Current issue occurs only in the case of 0 values, none zero
values with higher scale are still save in non scientific notation.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17899
**[Test build #95753 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95753/testReport)**
for PR 17899 at commit
Github user 10110346 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22350#discussion_r215598798
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
---
@@ -123,6 +123,9 @@ class
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22138
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user SongYadong opened a pull request:
https://github.com/apache/spark/pull/22348
Reduce unneeded operation in nextKeyValue process of parquet vectorized
record reader
## What changes were proposed in this pull request?
this PR do following in
Github user npoberezkin commented on the issue:
https://github.com/apache/spark/pull/22322
Yes, sure. I will do it soon (maybe next week)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22349
**[Test build #95751 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95751/testReport)**
for PR 22349 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22349
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22349
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22346
**[Test build #95743 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95743/testReport)**
for PR 22346 at commit
GitHub user 10110346 opened a pull request:
https://github.com/apache/spark/pull/22350
[SPARK-25356][SQL]Add Parquet block size option to SparkSQL configuration
## What changes were proposed in this pull request?
I think we should configure the Parquet buffer size
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22352
**[Test build #95755 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95755/testReport)**
for PR 22352 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22352
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22352
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22271
**[Test build #95747 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95747/testReport)**
for PR 22271 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22332
If that's easily worked around, let's not add this one. There are too many
APIs open now and we should rather try to reduce them.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22337
**[Test build #95748 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95748/testReport)**
for PR 22337 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22337
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22337
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/spark/pull/22318#discussion_r215571612
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -805,10 +807,10 @@ class Analyzer(
*
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/spark/pull/22318#discussion_r215571480
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala
---
@@ -23,12 +23,14 @@ package
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/spark/pull/22318#discussion_r215571667
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/AttributeMap.scala
---
@@ -23,12 +23,14 @@ package
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22138
**[Test build #95744 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95744/testReport)**
for PR 22138 at commit
Github user wmellouli closed the pull request at:
https://github.com/apache/spark/pull/22332
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user wmellouli commented on the issue:
https://github.com/apache/spark/pull/22332
PR closed: we can use select to add new columns in a user-defined position.
---
-
To unsubscribe, e-mail:
Github user ajithme commented on the issue:
https://github.com/apache/spark/pull/22277
Attaching a sql file to reproduce the issue and see the effect of PR :
[test.txt](https://github.com/apache/spark/files/2356468/test.txt)
### Without patch:
```
GitHub user WeichenXu123 opened a pull request:
https://github.com/apache/spark/pull/22349
[SPARK-25345][ML] Deprecate public APIs from ImageSchema
## What changes were proposed in this pull request?
Deprecate public APIs from ImageSchema.
## How was this patch
Github user peter-toth commented on a diff in the pull request:
https://github.com/apache/spark/pull/22318#discussion_r215571877
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -921,12 +924,18 @@ class Analyzer(
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22345
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22345
**[Test build #95745 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95745/testReport)**
for PR 22345 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22344
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/22138#discussion_r215591546
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala
---
@@ -0,0 +1,241 @@
+/*
+ *
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22344
**[Test build #95746 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95746/testReport)**
for PR 22344 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22351
**[Test build #95754 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95754/testReport)**
for PR 22351 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22344
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95746/
Test PASSed.
---
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22349#discussion_r215593840
--- Diff: python/pyspark/ml/image.py ---
@@ -20,6 +20,9 @@
An attribute of this module that contains the instance of
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22351
cc @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22351
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22349
**[Test build #95751 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95751/testReport)**
for PR 22349 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22138
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95744/
Test PASSed.
---
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22318
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22348
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22348
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ch0ice commented on the issue:
https://github.com/apache/spark/pull/181
This problem arose again for me, and I reproduced it when I converted byte
into protoBuf after redis checked the data.
The following code in the deserialization (Utils deserialize (value, Utils
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22345
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95745/
Test PASSed.
---
GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/22351
[MINOR][SQL] Add a debug log when a SQL text is used for a view
## What changes were proposed in this pull request?
This took me a while to debug and find out. Looks we better at least
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22351
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/22284
@cloud-fan shall we consider this for 2.4? I don't see any real
concern/comment about it, so I think it would be great if we can include it as
it is a bug.
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22316
Branch is cut out. Let's target 3.0.0
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22171
Hm, I don't think there's standard notation for numbers in CSV since the
datatype is specific to text if I remember the RFC 4180 correctly. Might be
good to double check.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22344
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22344
**[Test build #95741 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95741/testReport)**
for PR 22344 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22344
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95741/
Test FAILed.
---
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22332
Thanks, @wmellouli.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22349
**[Test build #95749 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95749/testReport)**
for PR 22349 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22349
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/22138#discussion_r215579562
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala
---
@@ -18,222 +18,247 @@
package
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/22138#discussion_r215583862
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/FetchedPoolSuite.scala
---
@@ -0,0 +1,299 @@
+/*
+ * Licensed
Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20999
> it seems currently credits can go to multiple developers;
Yes, but I don't know how to do that. Probably committers can do it in the
merging process, so I think the only thing I can do
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22349
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22349
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95751/
Test PASSed.
---
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22140#discussion_r215601350
--- Diff: python/pyspark/sql/tests.py ---
@@ -269,6 +269,10 @@ def test_struct_field_type_name(self):
struct_field = StructField("a",
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22140#discussion_r215601486
--- Diff: python/pyspark/sql/types.py ---
@@ -1397,6 +1397,8 @@ def _create_row_inbound_converter(dataType):
def _create_row(fields,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22140
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user wmellouli commented on the issue:
https://github.com/apache/spark/pull/22332
@HyukjinKwon even instead of using the actual method `withColumn(colName:
String, col: Column)` we can just add a column and select. The idea from this
PR is to add more power/flexibility to
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22171
BTW, was wondering if we should call the current way a .. kind of Java
standard? IIRC, Python's decimal representation doesn't use a scientific
notation by default. I thought this makes sense
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22295#discussion_r215556683
--- Diff: python/pyspark/sql/session.py ---
@@ -252,6 +252,16 @@ def newSession(self):
"""
return self.__class__(self._sc,
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22295#discussion_r215556819
--- Diff: python/pyspark/sql/session.py ---
@@ -252,6 +252,16 @@ def newSession(self):
"""
return self.__class__(self._sc,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22349
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22318
**[Test build #95750 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95750/testReport)**
for PR 22318 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22350
**[Test build #95752 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95752/testReport)**
for PR 22350 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22270
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22270
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95742/
Test PASSed.
---
Github user gaborgsomogyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/22138#discussion_r215594790
--- Diff:
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/FetchedPoolSuite.scala
---
@@ -0,0 +1,299 @@
+/*
+ * Licensed
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/22352
cc @cloud-fan
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/22352
[SPARK-25208][SQL][FOLLOW-UP] Reduce code size.
## What changes were proposed in this pull request?
When casting to decimal type, if `Cast.canNullSafeCastToDecimal()`,
overflow won't
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22318
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22318
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95750/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22348
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22271
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95747/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22271
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22271
**[Test build #95747 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95747/testReport)**
for PR 22271 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22350
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22350
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22270
**[Test build #95742 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95742/testReport)**
for PR 22270 at commit
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/22352
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AndrewKL commented on a diff in the pull request:
https://github.com/apache/spark/pull/22162#discussion_r215618109
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -969,6 +969,22 @@ class DatasetSuite extends QueryTest with
Github user rvesse commented on the issue:
https://github.com/apache/spark/pull/22215
Think this is pretty much ready to merge, can folks take another look when
they get chance
---
-
To unsubscribe, e-mail:
Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/22138#discussion_r215635068
--- Diff:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/InternalKafkaConsumerPool.scala
---
@@ -0,0 +1,241 @@
+/*
+ *
Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22165#discussion_r215635071
--- Diff: core/src/main/scala/org/apache/spark/BarrierCoordinator.scala ---
@@ -65,7 +65,7 @@ private[spark] class BarrierCoordinator(
//
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22352
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22352
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22144
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22354
**[Test build #95764 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95764/testReport)**
for PR 22354 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22354
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22355
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22355
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/22329
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17899
**[Test build #95753 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95753/testReport)**
for PR 17899 at commit
101 - 200 of 451 matches
Mail list logo
