[GitHub] spark issue #14481: [WIP][SPARK-16844][SQL] Generate code for sort based agg...

2016-11-25 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14481 @hvanhovell What's the status of this? If nobody takes this, I'll do. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #14481: [WIP][SPARK-16844][SQL] Generate code for sort based agg...

2016-11-25 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14481 @yucai okay, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16021: [SPARK-18593][SQL] JDBCRDD returns incorrect results for...

2016-11-27 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16021 @dongjoon-hyun Thanks for letting me know! I'm not familiar with a backport policy in spark though, if we can, I think it'd be better to do because it seems there are still some of users use spark

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-11-28 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz Thanks for your comment! okay, I'll fix in a that way. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-11-29 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #15980: [SPARK-18528][SQL] Fix a bug to initialise an iterator o...

2016-11-24 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/15980 @davies How about this fix? I think it seems to be difficult to put the `earlyStop` flag in `shouldStop()` because `shouldStop()` is used for a caller (an uppper `SparkPlan`) to exit a loop

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-11-28 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/16030 [SPARK-18108][SQL] Fix a bug to fail partition schema inference ## What changes were proposed in this pull request? This pr is to fix a bug to fail partition schema inference; ``` scala

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-11-28 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 This query passed in the released spark-2.0.2, so it seems this regression is involved with SPARK-18510. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-11-28 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz @tdas Could you check this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #13300: [SPARK-15463][SQL] support creating dataframe out of Dat...

2016-11-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13300 What's the status of this pr? It seems to be more natural that we implement `from_csv` in a similar way of `from_json` in https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org

[GitHub] spark issue #14812: [SPARK-17237][SQL] Remove backticks in a pivot result sc...

2016-11-18 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14812 @gatorsmile Do u have time to check this? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #14038: [SPARK-16317][SQL] Add a new interface to filter files i...

2016-11-18 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14038 @liancheng I'm not sure that the original motivation keeps alive in SPARK-16317 though, if I need to do something, please let me know. I made new code based on this pr (https://github.com/apache

[GitHub] spark pull request #15928: [SPARK-18478][SQL] Support codegen'd Hive UDFs

2016-11-18 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/15928 [SPARK-18478][SQL] Support codegen'd Hive UDFs ## What changes were proposed in this pull request? This pr is to support codegen'd Hive UDFs. ## How was this patch tested? Add tests

[GitHub] spark pull request #14039: [SPARK-15896][SQL] Clean up shuffle files just af...

2016-11-18 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/14039 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #15928: [SPARK-18478][SQL] Support codegen'd Hive UDFs

2016-11-18 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/15928 okay, thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #13300: [SPARK-15463][SQL] support creating dataframe out of Dat...

2016-11-21 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13300 I checked the feasibility to implement `from_json` in `sql.functions`; If we move csv parser code (`CSVReader`, `CSVOptions`, ...) from `o.a.s.sql.execution.datasources.csv

[GitHub] spark pull request #15945: [SPARK-12978][SQL] Merge unnecessary partial aggr...

2016-11-20 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/15945 [SPARK-12978][SQL] Merge unnecessary partial aggregates ## What changes were proposed in this pull request? This pr is to merge unnecessary partial aggregates if the inputs of aggregates

[GitHub] spark issue #14038: [SPARK-16317][SQL] Add a new interface to filter files i...

2016-11-20 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14038 yea, as for data files, it's okay to filter out '_' and '.'. But, the file pattens of metadata depend on file formats as suggested in https://github.com/apache/spark/blob/master/sql/core/src/main

[GitHub] spark issue #15945: [SPARK-12978][SQL] Merge unnecessary partial aggregates

2016-11-20 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/15945 @hvanhovell @cloud-fan I think this target might be 2.2.0, so could you check this after 2.1 is cut. Thanks! --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #15257: [SPARK-17683][SQL] Support ArrayType in Literal.apply

2016-11-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/15257 I think it is difficult to support Seq and JList because of type erasure. If it is possible to add a new interface below, we can support these types via `ScalaReflection.schemaFor[T

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-11-29 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 I'm looking into the failures. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHAR...

2016-12-08 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/16213 [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END to finish reading closed shards ## What changes were proposed in this pull request? This pr is to fix an issue occurred when resharding

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2016-12-08 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16213 I'm looking into the failure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHAR...

2016-12-09 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16213#discussion_r91829798 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala --- @@ -225,6 +225,74 @@ abstract class

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2016-12-09 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16213 Thanks for these comments! ya, I do not like this approach, too. But, since those who reshard streams always hit this issue and resharding is important for load-balancing in Kinesis streams

[GitHub] spark pull request #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHAR...

2016-12-09 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16213#discussion_r91829740 --- Diff: external/kinesis-asl/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/CheckpointerShim.java --- @@ -0,0 +1,43

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-12 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 What's the status of this pr? If I've something to do, plz let me know. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-13 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 okay, I'll try to fix in that way, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @cloud-fan okay, I updated the desc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a schema inconsistent bug that ma...

2016-12-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 oh, I wrongly wrote in an opposite way..., okay, fixed cc: @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a schema inconsistent bug that ma...

2016-12-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 Thanks! I found another wired behaviour related to this issue. Is this expected? or, should we fix? ``` scala> import org.apache.spark.sql.types._ scala> val schema = new Stru

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-06 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r91210953 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala --- @@ -68,9 +69,16 @@ private[kinesis] class

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-06 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r91208525 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala --- @@ -68,9 +69,16 @@ private[kinesis] class

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @cloud-fan Does the latest fix satisfy what you suggested? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @liancheng As for `DataFrameReader.dataSchema()` and `DataFrameReader.partitoinSchema()`, did you mean we add new interfaces there for users to set user-defined data and partition schema

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-14 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r92543342 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala --- @@ -969,4 +969,17 @@ class

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-15 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r92594503 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala --- @@ -49,9 +49,12 @@ case class HadoopFsRelation

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-15 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r92600244 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala --- @@ -49,10 +51,16 @@ case class HadoopFsRelation

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 okay --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-07 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r91419246 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala --- @@ -68,7 +68,16 @@ private[kinesis] class

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2016-12-08 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16213 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16114: [SPARK-18620][Streaming][Kinesis] Flatten input rates in...

2016-12-08 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16114 @brkyvz Could you also check this pr #16213? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request #10521: [SPARK-2827][GraphX] Add collectDegreeDist to com...

2017-01-10 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/10521 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #13736: [SPARK-12113][SQL] Add some timing metrics for blocking ...

2017-01-10 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13736 I think this pr got stale, so I'll close this, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #10521: [SPARK-2827][GraphX] Add collectDegreeDist to compute th...

2017-01-10 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/10521 I think the GraphX features seems freeze, so I'll close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16410: [SPARK-19005][SQL] Keep column ordering when a schema is...

2017-01-13 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16410 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16410: [SPARK-19005][SQL] Keep column ordering when a schema is...

2017-01-13 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16410 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16410: [SPARK-19005][SQL] Keep column ordering when a schema is...

2017-01-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16410 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14812: [SPARK-17237][SQL] Remove backticks in a pivot result sc...

2017-01-10 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14812 okay, thanks! I'll check again soon --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16565: [SPARK-17237][SQL] Remove backticks in a pivot result sc...

2017-01-12 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16565 okay! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16565: [SPARK-17237][SQL] Remove backticks in a pivot re...

2017-01-12 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/16565 [SPARK-17237][SQL] Remove backticks in a pivot result schema ## What changes were proposed in this pull request? Pivoting adds backticks (e.g. 3_count(\`c\`)) in column names and, in some cases

[GitHub] spark issue #15945: [SPARK-12978][SQL] Merge unnecessary partial aggregates

2017-01-12 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/15945 @cloud-fan How about this fix? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14812: [SPARK-17237][SQL] Remove backticks in a pivot result sc...

2017-01-12 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14812 okay, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #15945: [SPARK-12978][SQL] Merge unnecessary partial aggregates

2017-01-12 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/15945 @hvanhovell @cloud-fan Could you check this? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #14812: [SPARK-17237][SQL] Remove backticks in a pivot result sc...

2017-01-12 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14812 @gatorsmile okay, fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #15945: [SPARK-12978][SQL] Merge unnecessary partial aggr...

2017-01-12 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/15945#discussion_r95782977 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala --- @@ -90,18 +92,27 @@ class QueryExecution(val sparkSession

[GitHub] spark issue #16565: [SPARK-17237][SQL][Backport-2.0] Remove backticks in a p...

2017-01-13 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16565 @gatorsmile oh, I see. Is it okay to mix this pr with the fix of #15111? Would it be better to backport #15111 first then, backport this? --- If your project is set up for it, you can reply

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2017-01-12 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16213 @brkyvz I asked AWS guys and I found correct handling in this case (See: https://forums.aws.amazon.com/thread.jspa?threadID=244218). I fixed in this way and could you check this again? Thanks

[GitHub] spark issue #14038: [SPARK-16317][SQL] Add a new interface to filter files i...

2017-01-15 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/14038 @liancheng Could you check this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #11461: [SPARK-13607][SQL] Improve compression performanc...

2017-01-10 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/11461 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #13736: [SPARK-12113][SQL] Add some timing metrics for bl...

2017-01-10 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/13736 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #11461: [SPARK-13607][SQL] Improve compression performance for i...

2017-01-10 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/11461 I think this improvement is not always necessary, so I'll close this for now. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 The failure seems to be not related to this pr? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz yea, I think so, but the issue of column ordering seems to be out of scope in this original ticket. So, why don't we open a new jira and discuss there? --- If your project is set up

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r90754922 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala --- @@ -56,6 +56,31 @@ private[kinesis] class

[GitHub] spark issue #16114: [SPARK-18620][Streaming][Kinesis] Flatten input rates in...

2016-12-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16114 @srowen Do u know qualified maintainers on this component? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r90752283 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -189,7 +189,15 @@ case class DataSource

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r90758322 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala --- @@ -56,6 +56,27 @@ private[kinesis] class

[GitHub] spark issue #16114: [SPARK-18620][Streaming][Kinesis] Flatten input rates in...

2016-12-03 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16114 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-03 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r90758182 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala --- @@ -56,6 +56,27 @@ private[kinesis] class

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-01 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz Thanks! Does the latest fix apply your suggestion? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r90601340 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala --- @@ -221,6 +221,12 @@ private[kinesis] class

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r90619967 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala --- @@ -969,4 +969,15 @@ class

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r90620033 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -174,22 +185,18 @@ case class DataSource

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r90620001 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -174,22 +185,18 @@ case class DataSource

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16114#discussion_r90600966 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisRecordProcessor.scala --- @@ -56,6 +56,38 @@ private[kinesis] class

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz How should we handle this query? I think we have some overheads to check the constant value (`1480617712537`) in path matches all the data in DataFrame and to throw an exception if illegal

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

2016-12-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16030#discussion_r90620083 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala --- @@ -132,13 +132,24 @@ case class DataSource

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-11-29 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 @brkyvz How about this fix? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-04 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 okay, I'll do --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-11-30 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 This is not a bug in `VectorizedParquetReader ` as @liancheng said, and the root cause is that wrongly inferred types are passed into the reader in `VectorizedParquetReader#initBatch` https

[GitHub] spark pull request #16114: [SPARK-18620][Streaming][Kinesis] Flatten input r...

2016-12-01 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/16114 [SPARK-18620][Streaming][Kinesis] Flatten input rates in timeline for streaming + kinesis ## What changes were proposed in this pull request? This pr is to make input rates in timeline more

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

2016-12-01 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 Thanks your review, @brkyvz ! I'm checking your comments now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #16319: [WiP][SPARK-18699] SQL - parsing CSV should retur...

2016-12-18 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16319#discussion_r92959056 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala --- @@ -246,9 +246,9 @@ private[csv] object

[GitHub] spark issue #16410: [SPARK-19005][SQL] Keep column ordering when a schema is...

2016-12-27 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16410 Aha, okay and I'll fix that way. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #16410: [SPARK-19005][SQL] Keep column ordering when a sc...

2016-12-26 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/16410 [SPARK-19005][SQL] Keep column ordering when a schema is explicitly specified ## What changes were proposed in this pull request? This pr is to keep column ordering when a schema is explicitly

[GitHub] spark issue #16410: [SPARK-19005][SQL] Keep column ordering when a schema is...

2016-12-26 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16410 I'm looking into the failure... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16410: [SPARK-19005][SQL] Keep column ordering when a schema is...

2016-12-27 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16410 This fix change some existing behaviour in datasource. For instance, ``` scala> sql("""CREATE TABLE testTable(a INT, b INT, c INT, d INT) USING PARQUET PARTITIONED BY (b

[GitHub] spark pull request #16319: [SPARK-18699] SQL - parsing CSV should return nul...

2016-12-16 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/16319#discussion_r92915632 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala --- @@ -246,9 +246,9 @@ private[csv] object

[GitHub] spark issue #16319: [SPARK-18699] SQL - parsing CSV should return null for c...

2016-12-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16319 @HyukjinKwon Thanks for your pinging! I left some comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a schema inconsistent bug that ma...

2016-12-16 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 okay! I'll make a JIRA later and thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15928: [SPARK-18478][SQL] Support codegen'd Hive UDFs

2017-01-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/15928 @hvanhovell Could you check this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16565: [SPARK-17237][SPARK-17458][SQL][Backport-2.0] Preserve a...

2017-01-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16565 Okay and thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark pull request #16565: [SPARK-17237][SPARK-17458][SQL][Backport-2.0] Pre...

2017-01-14 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/16565 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #16213: [SPARK-18020][Streaming][Kinesis] Checkpoint SHARD_END t...

2017-01-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16213 @brkyvz ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16565: [SPARK-17237][SQL][Backport-2.0] Remove backticks in a p...

2017-01-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16565 @gatorsmile ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #16410: [SPARK-19005][SQL] Keep column ordering when a schema is...

2017-01-14 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/16410 I looked around the code and then I though this is an expected behaviour, so I'll close this. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request #16410: [SPARK-19005][SQL] Keep column ordering when a sc...

2017-01-14 Thread maropu
Github user maropu closed the pull request at: https://github.com/apache/spark/pull/16410 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

<    5   6   7   8   9   10   11   12   13   14   >