[GitHub] spark pull request: [SPARK-11474][SQL]change fetchSize to fetchsiz...

2015-11-04 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/9473 [SPARK-11474][SQL]change fetchSize to fetchsize In DefaultDataSource.scala, it has override def createRelation( sqlContext: SQLContext, parameters: Map[String, String]): BaseRelation

[GitHub] spark pull request: [SPARK-11474]Options to jdbc load are lower ca...

2015-11-04 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/9461 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-11474]Options to jdbc load are lower ca...

2015-11-04 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9461#issuecomment-153832023 I am closing this pull request as i think i reused my previous branch.. i would like to start clean. I will open a new pull request shortly. Sorry for the confusion

[GitHub] spark pull request: Spark 11474

2015-11-03 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/9461 Spark 11474 You can merge this pull request into a Git repository by running: $ git pull https://github.com/huaxingao/spark spark_11474 Alternatively you can review and apply these changes

[GitHub] spark pull request: [SPARK-8386] [SQL]add write.mode for insertInt...

2015-10-14 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9042#issuecomment-148122772 I looked the error log, it failed at fetching changes from the remote Git repository. Does it mean something is wrong with my pull request and I need to do a new

[GitHub] spark pull request: [SPARK-8386] [SQL]add write.mode for insertInt...

2015-10-08 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/9042 [SPARK-8386] [SQL]add write.mode for insertIntoJDBC when the parm overwrite is false the fix is for jira https://issues.apache.org/jira/browse/SPARK-8386 You can merge this pull request

[GitHub] spark pull request: [SPARK-8386] [SQL]add write.mode for insertInt...

2015-10-12 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9042#issuecomment-147592598 Has the test result come back yet? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-11778:parse table name before it is pass...

2015-11-17 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9773#issuecomment-157490181 hiveContext.table("db_name.table") works but hiveContext.read.table("db_name.table") throws an org.apache.spark.sql.catalyst.analysis.

[GitHub] spark pull request: SPARK-11778:parse table name before it is pass...

2015-11-17 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/9773 SPARK-11778:parse table name before it is passed to lookupRelation You can merge this pull request into a Git repository by running: $ git pull https://github.com/huaxingao/spark spark

[GitHub] spark pull request: [SPARK-11788][SQL] surround timestamp/date val...

2015-11-27 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9872#issuecomment-160182654 Fixed title and scala style problem. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-12088][SQL]check connection.isClosed be...

2015-12-02 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10095 [SPARK-12088][SQL]check connection.isClosed before calling connection… In Java Spec java.sql.Connection, it has boolean getAutoCommit() throws SQLException Throws: SQLException

[GitHub] spark pull request: [SPARK-12088][SQL]check connection.isClosed be...

2015-12-02 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10095#issuecomment-161219510 I guess I don't need to provide a regression test. Currently, in JDBCSuite, there are lots of Warning as the following, the Warning will be gone after the fix

[GitHub] spark pull request: [SPARK-12391][SQL]JDBC OR operator push down

2015-12-16 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10347 [SPARK-12391][SQL]JDBC OR operator push down You can merge this pull request into a Git repository by running: $ git pull https://github.com/huaxingao/spark spark-12391 Alternatively you

[GitHub] spark pull request: [SPARK-12387][SQL]JDBC IN operator push down

2015-12-16 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10345 [SPARK-12387][SQL]JDBC IN operator push down Will push down SQL IN operator such as the following to JDBC datasource SELECT column_name(s) FROM table_name WHERE column_name

[GitHub] spark pull request: [SPARK-12270][SQL]remove empty space after get...

2015-12-15 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10262#issuecomment-164680145 @andrewor14 Thanks a lot for your comment. I will change to what you suggested. In the same method, case DecimalConversion has the similar code. Shall I

[GitHub] spark pull request: [SPARK-12387][SQL]JDBC IN operator push down

2015-12-17 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/10345#discussion_r47965769 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala --- @@ -269,6 +269,13 @@ private[sql] class JDBCRDD

[GitHub] spark pull request: [SPARK-12270][SQL]remove empty space after get...

2015-12-10 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10262 [SPARK-12270][SQL]remove empty space after getString from database {code} conn.prepareStatement( "create table people (name char(32)").exe

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2016-01-04 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10505#issuecomment-168901512 @rxin I am not sure if my approach is OK. Could you please take a quick look when you have time and let me know what you think? Thank you very much for your

[GitHub] spark pull request: [SPARK-12391][SQL]JDBC OR operator push down

2015-12-18 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/10347 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-12409][SQL]JDBC AND operator push down

2015-12-18 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/10369 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-12409][SQL]JDBC AND operator push down

2015-12-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10369#issuecomment-165850180 Close for now. Will put the filter changes in one PR so it's easier to merge. Sorry for the inconvenience. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-12409][SQL]JDBC AND operator push down

2015-12-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10369#issuecomment-165848876 Sorry for the trouble. I should have everything in one PR. Will do it now. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-12387][SQL]JDBC IN operator push down

2015-12-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10345#issuecomment-165849528 Close for now. Will put the filter changes in one PR so it's easier to merge --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-12387][SQL]JDBC IN operator push down

2015-12-18 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/10345 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-12391][SQL]JDBC OR operator push down

2015-12-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10347#issuecomment-165849800 Close for now. Will put the filter changes in one PR so it's easier to merge. Sorry for the inconvenience. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-12270][SQL]remove empty space after get...

2015-12-19 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10262#issuecomment-166033287 @yhuai JDBCSuite uses H2 database. It seems that for char(n) data type, either H2 database doesn't pad, or the H2 JDBC driver already trims the empty space

[GitHub] spark pull request: [SPARK-12409][SQL]add filter (IN, AND, OR)

2015-12-18 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10386 [SPARK-12409][SQL]add filter (IN, AND, OR) push filters IN, ADD, OR to JDBC layer. You can merge this pull request into a Git repository by running: $ git pull https://github.com/huaxingao

[GitHub] spark pull request: [SPARK-12459][SQL]add ExpressionDescription to...

2015-12-23 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10460 [SPARK-12459][SQL]add ExpressionDescription to string functions You can merge this pull request into a Git repository by running: $ git pull https://github.com/huaxingao/spark spark-12459

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2015-12-29 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10505 [SPARK-12506][SQL]push down WHERE clause arithmetic operator to JDBC … …layer For arithmetic operator in WHERE clause such as select * from table where c1 + c2 > 10 Curren

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2015-12-29 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10505#issuecomment-167746195 I only added + operator for now. If the change is accepted, I will also add -,* and /. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-12391][SQL]JDBC OR operator push down

2015-12-17 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/10347#discussion_r47948040 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala --- @@ -288,6 +288,8 @@ private[sql] class JDBCRDD

[GitHub] spark pull request: [SPARK-12409][SQL]JDBC AND operator push down

2015-12-17 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10369 [SPARK-12409][SQL]JDBC AND operator push down For simple AND such as select * from test where THEID = 1 AND NAME = 'fred', The filters pushed down to JDBC layers are EqualTo(THEID,1

[GitHub] spark pull request: [SPARK-11788][SQL]:surround timestamp/date val...

2015-11-24 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9872#issuecomment-159375392 @JoshRosen Regression test added. Could you please take a look? Thanks a lot for your help!! --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-11778] [SQL]:parse table name before it...

2015-11-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9773#issuecomment-157915848 test case added. Could you please take a look? Thanks a lot!! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-11788][SQL]:surround timestamp/date val...

2015-11-20 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/9872 [SPARK-11788][SQL]:surround timestamp/date value with quotes You can merge this pull request into a Git repository by running: $ git pull https://github.com/huaxingao/spark spark-11788

[GitHub] spark pull request: [SPARK-11788][SQL]:surround timestamp/date val...

2015-11-20 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9872#issuecomment-158545092 Sure. Will add a test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-11778:parse table name before it is pass...

2015-11-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9773#issuecomment-157865029 I will add a test case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-11778][SQL]:add regression test

2015-11-21 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/9890 [SPARK-11778][SQL]:add regression test Fix regression test for SPARK-11778. @marmbrus Could you please take a look? Thank you very much!! You can merge this pull request into a Git

[GitHub] spark pull request: [SPARK-11778][SQL]:add regression test

2015-11-23 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/9890#issuecomment-159143868 I couldn't figure out why the tests failed, but it doesn't seem to me that my new test suite caused the failure. Is it OK to have a retest? Thanks a lot

[GitHub] spark pull request: [SPARK-12270][SQL]remove empty space after get...

2016-01-11 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10262#issuecomment-170701187 @yhuai Sorry for the late reply. I waited for my coworker Luciano to come back from vacation today to check with him about his DB2 docker test status. He has

[GitHub] spark issue #13287: [SPARK-15491][SQL]fix assertion failure for JDBC DataFra...

2016-06-02 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/13287 @rxin gentle ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #13492: [SPARK-15749][SQL]make the error message more meaningful

2016-06-03 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/13492 retest please. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #13492: [SPARK-15749][SQL]make the error message more mea...

2016-06-03 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/13492 [SPARK-15749][SQL]make the error message more meaningful ## What changes were proposed in this pull request? For table test1 (C1 varchar (10), C2 varchar (10)), when I insert a row

[GitHub] spark pull request: [SPARK-15491][SQL]fix assertion failure for JD...

2016-05-25 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/13287#issuecomment-221617823 @rxin Sorry, I didn't notice your comment until this morning. Reformatted the description. Thanks! --- If your project is set up for it, you can reply

[GitHub] spark issue #13492: [SPARK-15749][SQL]make the error message more meaningful

2016-06-14 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/13492 @andrewor14 I am not sure why the test failed, but it doesn't seem to be related to my change. Could you please start another test? Thanks a lot!! --- If your project is set up

[GitHub] spark pull request: [SPARK-15491][SQL]fix assertion failure for JD...

2016-05-27 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/13287#issuecomment-21356 @rxin Thanks for your comment. When I wrote the test, I found there is one more place I need to change. So there are two places that have problems: 1

[GitHub] spark issue #13492: [SPARK-15749][SQL]make the error message more meaningful

2016-06-16 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/13492 @andrewor14 Thanks a lot for merging in the change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2016-01-13 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10750#issuecomment-171498490 @viirya I changed the code based on your suggestion. Could you please review again? Thanks a lot for your help!! --- If your project is set up

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2016-01-13 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/10750 [SPARK-12506][SQL]push down WHERE clause arithmetic operator to JDBC … …layer For arithmetic operator in WHERE clause such as select * from table where c1 + c2 > 10 Curren

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2016-01-13 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/10505 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2016-01-13 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/10505#discussion_r49678184 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala --- @@ -485,6 +486,74 @@ private[sql] object

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-06 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/11104 [SPARK-13186][Streaming]Migrate away from SynchronizedMap trait SynchronizedMap in package mutable is deprecated: Synchronization via traits is deprecated as it is inherently unreliable. Change

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-06 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11104#issuecomment-180883943 @holdenk Could you please review? Thanks!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-07 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/11104#discussion_r52126950 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala --- @@ -30,6 +30,9 @@ import org.apache.spark.{SparkConf

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-08 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11104#issuecomment-181664339 Sorry for the file line length problem. Fixed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-08 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/11104#discussion_r52228031 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala --- @@ -65,12 +67,14 @@ class KafkaStreamSuite extends

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-08 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11104#issuecomment-181528520 @holdenk Could you please review one more time? I changed to java api except the getOrElseUpdate in KafkaStreamSuite.scala. I can't find a java equivalent

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-12 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11104#issuecomment-183434809 @srowen Will do. I have my local branch messed up. If i can't figure out how to fix it, I will close this PR and submit a new one. Also, one of the python

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-08 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11104#issuecomment-181633831 @zsxwing Thanks for the comments. I didn't see a PR for removing SynchronizedSet. I will work on this. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-08 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/11104#discussion_r52250939 --- Diff: external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala --- @@ -65,12 +67,14 @@ class KafkaStreamSuite extends

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-08 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11104#issuecomment-181648992 Fixed the problems. Thank you all very much for your help!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-12506][SPARK-12126][SQL]use CatalystSca...

2016-02-01 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/11005 [SPARK-12506][SPARK-12126][SQL]use CatalystScan for JDBCRelation As suggested https://issues.apache.org/jira/browse/SPARK-9182?focusedCommentId=15031526page

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2016-02-01 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/10750 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-12506][SQL]push down WHERE clause arith...

2016-02-01 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10750#issuecomment-178119526 @viirya @HyukjinKwon @rxin Thank you all very much for your comments. I will change JDBCRelation to implement CatalystScan, and then directly access Catalyst

[GitHub] spark pull request: [SPARK-13186][Streaming]migrate away from Sync...

2016-02-22 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11250#issuecomment-187305582 @srowen @holdenk Thank you very much for your help!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-18 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/11104 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: [SPARK-13186][Streaming]Migrate away from Sync...

2016-02-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11104#issuecomment-185604993 @srowen @holdenk I will close this PR and submit a new one. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-13186][Streaming]migrate away from Sync...

2016-02-18 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/11250 [SPARK-13186][Streaming]migrate away from SynchronizedMap trait SynchronizedMap in package mutable is deprecated: Synchronization via traits is deprecated as it is inherently unreliable. Change

[GitHub] spark pull request: [SPARK-13186][Streaming]migrate away from Sync...

2016-02-18 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/11250#issuecomment-185608162 @srowen @holdenk Could you please take a look of this PR? I ran the python streaming test cleanly on my local before I submitted the PR. Thanks

[GitHub] spark pull request: [SPARK-13186][Streaming]migrate away from Sync...

2016-02-18 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/11250#discussion_r53430913 --- Diff: extras/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala --- @@ -241,13 +243,13 @@ abstract class

[GitHub] spark pull request: [SPARK-13186][Streaming]migrate away from Sync...

2016-02-21 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/11250#discussion_r53563513 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -163,8 +166,11 @@ class FileInputDStream[K, V, F

[GitHub] spark pull request: [SPARK-12270][SQL]remove empty space after get...

2016-04-25 Thread huaxingao
Github user huaxingao commented on the pull request: https://github.com/apache/spark/pull/10262#issuecomment-214437787 @HyukjinKwon I will continue working on this and finish the work this week. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-15491][SQL]fix assertion failure for JD...

2016-05-24 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/13287 [SPARK-15491][SQL]fix assertion failure for JDBC DataFrame to JSON ## What changes were proposed in this pull request? in TreeNode.scala parseToJson, it has case p: Product =>

[GitHub] spark pull request #14535: [SPARK-16946][SQL]throw Exception if saveAsTable[...

2016-08-08 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/14535 [SPARK-16946][SQL]throw Exception if saveAsTable[apend] has different… ## What changes were proposed in this pull request? In HiveContext, if saveAsTable[append] has different number

[GitHub] spark pull request #10262: [SPARK-12270][SQL]remove empty space after getStr...

2017-02-09 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/10262 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #16175: [SPARK-17460][SQL]check if statistics.sizeInBytes...

2016-12-07 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/16175#discussion_r91345742 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala --- @@ -1110,6 +1110,16 @@ class DatasetSuite extends QueryTest

[GitHub] spark issue #16175: [SPARK-17460][SQL]check if statistics.sizeInBytes >=0 in...

2016-12-07 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/16175 @gatorsmile Thanks a lot for reviewing this. Sorry I just saw your last comment after I pushed the change. Will make more changes for other potential overflow issues. --- If your project

[GitHub] spark issue #16175: [SPARK-17460][SQL]check if statistics.sizeInBytes >=0 in...

2016-12-06 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/16175 @gatorsmile Could you please take a look when you have time? Thanks a lot!! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request #16175: [SPARK-17460][SQL]check if statistics.sizeInBytes...

2016-12-06 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/16175 [SPARK-17460][SQL]check if statistics.sizeInBytes >=0 in canBroadcast ## What changes were proposed in this pull request? 1. In SparkStrategies.canBroadcast, I will add the ch

[GitHub] spark pull request #16175: [SPARK-17460][SQL]check if statistics.sizeInBytes...

2016-12-09 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/16175#discussion_r91765482 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -115,7 +115,8 @@ abstract class SparkStrategies extends

[GitHub] spark pull request #14535: [SPARK-16946][SQL]throw Exception if saveAsTable[...

2017-06-20 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/14535 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #14535: [SPARK-16946][SQL]throw Exception if saveAsTable[apend] ...

2017-06-20 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/14535 @gatorsmile Sorry for the late response. I just came back from China. This is not an issue any more. I will close this PR. --- If your project is set up for it, you can reply

[GitHub] spark issue #13287: [SPARK-15491][SQL]fix assertion failure for JDBC DataFra...

2017-06-20 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/13287 @gatorsmile Sorry for the late response. I just came back from China. For a class that has two level of constructor parameters, e.g. private[sql] case class JDBCRelation

[GitHub] spark pull request #13287: [SPARK-15491][SQL]fix assertion failure for JDBC ...

2017-06-21 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/13287#discussion_r123368171 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -594,7 +596,9 @@ abstract class TreeNode[BaseType

[GitHub] spark pull request #13287: [SPARK-15491][SQL]fix assertion failure for JDBC ...

2017-06-23 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/13287#discussion_r123818099 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala --- @@ -594,7 +596,9 @@ abstract class TreeNode[BaseType

[GitHub] spark pull request #13287: [SPARK-15491][SQL]fix assertion failure for JDBC ...

2017-06-23 Thread huaxingao
Github user huaxingao closed the pull request at: https://github.com/apache/spark/pull/13287 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request #19256: [SPARK-21338][SQL]implement isCascadingTruncateTa...

2017-09-16 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/19256 [SPARK-21338][SQL]implement isCascadingTruncateTable() method in Aggr… …egatedDialect ## What changes were proposed in this pull request

[GitHub] spark pull request #19256: [SPARK-21338][SQL]implement isCascadingTruncateTa...

2017-09-17 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19256#discussion_r139313120 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala --- @@ -41,4 +41,8 @@ private class AggregatedDialect(dialects: List

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/19256 Thanks @gatorsmile I will change both the implementation and the PR title. --- - To unsubscribe, e-mail: reviews

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-17 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/19256 Thanks @gatorsmile Does the following logic look good to you? ``` if(any dialect's isCascadingTruncateTable returns true) return Some(true) else if (any dialect's

[GitHub] spark issue #19256: [SPARK-21338][SQL]implement isCascadingTruncateTable() m...

2017-09-19 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/19256 @gatorsmile Thanks a lot for your help!!! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19496: [SPARK-22271][SQL]mean overflows and returns null...

2017-10-15 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19496#discussion_r144732734 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2103,4 +2103,13 @@ class DataFrameSuite extends QueryTest

[GitHub] spark pull request #19496: [SPARK-22271][SQL]mean overflows and returns null...

2017-10-13 Thread huaxingao
GitHub user huaxingao opened a pull request: https://github.com/apache/spark/pull/19496 [SPARK-22271][SQL]mean overflows and returns null for some decimal variables ## What changes were proposed in this pull request? In Average.scala, it has ``` override

[GitHub] spark pull request #19496: [SPARK-22271][SQL]mean overflows and returns null...

2017-10-17 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19496#discussion_r145182120 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala --- @@ -2103,4 +2103,13 @@ class DataFrameSuite extends QueryTest

[GitHub] spark issue #19496: [SPARK-22271][SQL]mean overflows and returns null for so...

2017-10-17 Thread huaxingao
Github user huaxingao commented on the issue: https://github.com/apache/spark/pull/19496 @gatorsmile Thank you very much for your help! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark pull request #19715: [SPARK-22397][ML]add multiple columns support to ...

2017-11-12 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19715#discussion_r150450334 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala --- @@ -146,4 +146,172 @@ class QuantileDiscretizerSuite

[GitHub] spark pull request #19715: [SPARK-22397][ML]add multiple columns support to ...

2017-11-12 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19715#discussion_r150451065 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala --- @@ -146,4 +146,172 @@ class QuantileDiscretizerSuite

[GitHub] spark pull request #19715: [SPARK-22397][ML]add multiple columns support to ...

2017-11-12 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19715#discussion_r150450151 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala --- @@ -104,7 +126,8 @@ private[feature] trait

[GitHub] spark pull request #19715: [SPARK-22397][ML]add multiple columns support to ...

2017-11-12 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19715#discussion_r150450280 --- Diff: mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala --- @@ -146,4 +146,172 @@ class QuantileDiscretizerSuite

[GitHub] spark pull request #19715: [SPARK-22397][ML]add multiple columns support to ...

2017-11-12 Thread huaxingao
Github user huaxingao commented on a diff in the pull request: https://github.com/apache/spark/pull/19715#discussion_r150450222 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala --- @@ -129,34 +152,95 @@ final class QuantileDiscretizer @Since

  1   2   3   4   >