[jira] [Updated] (SPARK-27820) case insensitive resolver should be used in GetMapValue

2019-05-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-27820: - Description: When extracting a key value from a MapType, it calls GetMapValue

[jira] [Updated] (SPARK-27820) case insensitive resolver should be used in GetMapValue

2019-05-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-27820: - Description: When extracting a key value from a MapType, it calls GetMapValue

[jira] [Updated] (SPARK-27820) case insensitive resolver should be used in GetMapValue

2019-05-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-27820: - Description: When extracting a key value from a MapType, it calls GetMapValue

[jira] [Created] (SPARK-27820) case insensitive resolver should be used in GetMapValue

2019-05-23 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-27820: Summary: case insensitive resolver should be used in GetMapValue Key: SPARK-27820 URL: https://issues.apache.org/jira/browse/SPARK-27820 Project: Spark

[jira] [Updated] (SPARK-23961) pyspark toLocalIterator throws an exception

2018-04-11 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-23961: - Description: Given a dataframe and use toLocalIterator. If we do not consume all records, it

[jira] [Updated] (SPARK-23961) pyspark toLocalIterator throws an exception

2018-04-11 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-23961: - Issue Type: Bug (was: Improvement) > pyspark toLocalIterator throws an exception >

[jira] [Created] (SPARK-23961) pyspark toLocalIterator throws an exception

2018-04-11 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-23961: Summary: pyspark toLocalIterator throws an exception Key: SPARK-23961 URL: https://issues.apache.org/jira/browse/SPARK-23961 Project: Spark Issue Type:

[jira] [Commented] (SPARK-23216) Multiclass LogisticRegression could have methods like NCE, NEG, Hierarchical SoftMax, Blackout or IS

2018-01-26 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341019#comment-16341019 ] Michel Lemay commented on SPARK-23216: -- Another interesting topic on the subject: Efficient softmax

[jira] [Created] (SPARK-23216) Multiclass LogisticRegression could have methods like NCE, NEG, Hierarchical SoftMax, Blackout or IS

2018-01-25 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-23216: Summary: Multiclass LogisticRegression could have methods like NCE, NEG, Hierarchical SoftMax, Blackout or IS Key: SPARK-23216 URL:

[jira] [Commented] (SPARK-13493) json to DataFrame to parquet does not respect case sensitiveness

2017-12-06 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281112#comment-16281112 ] Michel Lemay commented on SPARK-13493: -- Here we decided to snake_case everything and leave case

[jira] [Commented] (SPARK-12216) Spark failed to delete temp directory

2017-09-14 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166117#comment-16166117 ] Michel Lemay commented on SPARK-12216: -- In my case, what prevent the temp folder to be deleted is a

[jira] [Commented] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-14 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049100#comment-16049100 ] Michel Lemay commented on SPARK-21021: -- Indeed, the code is not broken so it's not a bug. However,

[jira] [Comment Edited] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-13 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047849#comment-16047849 ] Michel Lemay edited comment on SPARK-21021 at 6/13/17 1:06 PM: --- Yes, as a

[jira] [Comment Edited] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-13 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047849#comment-16047849 ] Michel Lemay edited comment on SPARK-21021 at 6/13/17 1:04 PM: --- Yes, as a

[jira] [Comment Edited] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-13 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047849#comment-16047849 ] Michel Lemay edited comment on SPARK-21021 at 6/13/17 1:04 PM: --- Yes, as a

[jira] [Comment Edited] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-13 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047849#comment-16047849 ] Michel Lemay edited comment on SPARK-21021 at 6/13/17 1:01 PM: --- Yes, as a

[jira] [Commented] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-13 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047849#comment-16047849 ] Michel Lemay commented on SPARK-21021: -- Yes, as a workaround, we do a

[jira] [Updated] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-08 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-21021: - Description: When reading back a partitioned parquet folder, column order gets messed up.

[jira] [Created] (SPARK-21021) Reading partitioned parquet does not respect specified schema column order

2017-06-08 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-21021: Summary: Reading partitioned parquet does not respect specified schema column order Key: SPARK-21021 URL: https://issues.apache.org/jira/browse/SPARK-21021 Project:

[jira] [Commented] (SPARK-20660) Not able to merge Dataframes with different column orders

2017-05-24 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1602#comment-1602 ] Michel Lemay commented on SPARK-20660: -- In my opinion, two schema should be considered the same if

[jira] [Created] (SPARK-20724) spark-submit verbose mode should list default settings values

2017-05-12 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-20724: Summary: spark-submit verbose mode should list default settings values Key: SPARK-20724 URL: https://issues.apache.org/jira/browse/SPARK-20724 Project: Spark

[jira] [Commented] (SPARK-13486) Move SQLConf into an internal package

2017-05-11 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006424#comment-16006424 ] Michel Lemay commented on SPARK-13486: -- This is troubling that this class isn't accessible outside

[jira] [Updated] (SPARK-20660) Not able to merge Dataframes with different column orders

2017-05-08 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-20660: - Description: Union on two dataframes with different column orders is not supported and lead to

[jira] [Updated] (SPARK-20660) Not able to merge Dataframes with different column orders

2017-05-08 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-20660: - Description: Union on two dataframes with different column orders is not supported and lead to

[jira] [Updated] (SPARK-20660) Not able to merge Dataframes with different column orders

2017-05-08 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-20660: - Description: Union on two dataframes with different column orders is not supported and lead to

[jira] [Created] (SPARK-20660) Not able to merge Dataframes with different column orders

2017-05-08 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-20660: Summary: Not able to merge Dataframes with different column orders Key: SPARK-20660 URL: https://issues.apache.org/jira/browse/SPARK-20660 Project: Spark

[jira] [Commented] (SPARK-16938) Cannot resolve column name after a join

2017-03-27 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943382#comment-15943382 ] Michel Lemay commented on SPARK-16938: -- I just stumbled upon a similar issue with 'union' on two

[jira] [Comment Edited] (SPARK-20061) Reading a file with colon (:) from S3 fails with URISyntaxException

2017-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938152#comment-15938152 ] Michel Lemay edited comment on SPARK-20061 at 3/23/17 11:54 AM: I don't

[jira] [Commented] (SPARK-20061) Reading a file with colon (:) from S3 fails with URISyntaxException

2017-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938152#comment-15938152 ] Michel Lemay commented on SPARK-20061: -- I don't know about hdfs api but we've never had any issues

[jira] [Created] (SPARK-20061) Reading a file with colon (:) from S3 fails with URISyntaxException

2017-03-22 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-20061: Summary: Reading a file with colon (:) from S3 fails with URISyntaxException Key: SPARK-20061 URL: https://issues.apache.org/jira/browse/SPARK-20061 Project: Spark

[jira] [Updated] (SPARK-19980) Basic Dataset transformation on POJOs does not preserves nulls.

2017-03-16 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-19980: - Description: Applying an identity map transformation on a statically typed Dataset with a POJO

[jira] [Updated] (SPARK-19980) Basic Dataset transformation on POJOs does not preserves nulls.

2017-03-16 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-19980: - Description: Applying an identity map transformation on a statically typed Dataset with a POJO

[jira] [Created] (SPARK-19980) Basic Dataset transformation on POJOs does not preserves nulls.

2017-03-16 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-19980: Summary: Basic Dataset transformation on POJOs does not preserves nulls. Key: SPARK-19980 URL: https://issues.apache.org/jira/browse/SPARK-19980 Project: Spark

[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-19 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761619#comment-15761619 ] Michel Lemay commented on SPARK-18648: -- I tried that approach before (spark.driver.extraClassPath)

[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-16 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755317#comment-15755317 ] Michel Lemay commented on SPARK-18648: -- It''s related because the path given to textFile is on a

[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-16 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755102#comment-15755102 ] Michel Lemay commented on SPARK-18648: -- it falls in case 2: I can import symbols, but

[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-16 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754991#comment-15754991 ] Michel Lemay commented on SPARK-18648: -- Sorry if I confused things up a little bit. Here it is:

[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-15 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752446#comment-15752446 ] Michel Lemay commented on SPARK-18648: -- I've also tried various combinations of file and slashes and

[jira] [Comment Edited] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-15 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752409#comment-15752409 ] Michel Lemay edited comment on SPARK-18648 at 12/15/16 8:22 PM: Well,

[jira] [Comment Edited] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-15 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752409#comment-15752409 ] Michel Lemay edited comment on SPARK-18648 at 12/15/16 8:21 PM: Well,

[jira] [Reopened] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-15 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay reopened SPARK-18648: -- Well, loading a jar like that is fine and make symbols available to the REPL. However, it

[jira] [Closed] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-15 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay closed SPARK-18648. Resolution: Not A Bug > spark-shell --jars option does not add jars to classpath on windows >

[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-15 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752140#comment-15752140 ] Michel Lemay commented on SPARK-18648: -- In order to make it work, one must use forward slash in the

[jira] [Commented] (SPARK-17262) Spark SizeEstimator does not ignore transient fields in java classes when calculates class size

2016-12-14 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748638#comment-15748638 ] Michel Lemay commented on SPARK-17262: -- In that case, It makes no sense to keep `val predictions`

[jira] [Commented] (SPARK-17262) Spark SizeEstimator does not ignore transient fields in java classes when calculates class size

2016-12-14 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748307#comment-15748307 ] Michel Lemay commented on SPARK-17262: -- I agree. As a realworld example of this, consider KMeans:

[jira] [Commented] (SPARK-18808) ml.KMeansModel.transform is very inefficient

2016-12-09 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736254#comment-15736254 ] Michel Lemay commented on SPARK-18808: -- Subclassing/overriding/adding methods in KMeans/Model is a

[jira] [Created] (SPARK-18808) ml.KMeansModel.transform is very inefficient

2016-12-09 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-18808: Summary: ml.KMeansModel.transform is very inefficient Key: SPARK-18808 URL: https://issues.apache.org/jira/browse/SPARK-18808 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-12-05 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722546#comment-15722546 ] Michel Lemay commented on SPARK-18648: -- Doing the same in Spark 1.6.2 was working correctly. This

[jira] [Updated] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-11-30 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-18648: - Description: I can't import symbols from command line jars when in the shell: Adding jars via

[jira] [Updated] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-11-30 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-18648: - Description: I can't import symbols from command line jars when in the shell: Adding jars via

[jira] [Created] (SPARK-18648) spark-shell --jars option does not add jars to classpath on windows

2016-11-30 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-18648: Summary: spark-shell --jars option does not add jars to classpath on windows Key: SPARK-18648 URL: https://issues.apache.org/jira/browse/SPARK-18648 Project: Spark

[jira] [Updated] (SPARK-14956) Spark dependencies conflicts

2016-04-27 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-14956: - Description: Since Spark lives with ages-old dependencies, it's all too often a problem that we

[jira] [Commented] (SPARK-14956) Spark dependencies conflicts

2016-04-27 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15260161#comment-15260161 ] Michel Lemay commented on SPARK-14956: -- Of course, this must be painful as well for Spark

[jira] [Commented] (SPARK-14956) Spark dependencies conflicts

2016-04-27 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15260087#comment-15260087 ] Michel Lemay commented on SPARK-14956: -- Here, we use Apache Felix for this purpose. I'm no expert

[jira] [Created] (SPARK-14956) Spark dependencies conflicts

2016-04-27 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-14956: Summary: Spark dependencies conflicts Key: SPARK-14956 URL: https://issues.apache.org/jira/browse/SPARK-14956 Project: Spark Issue Type: Brainstorming

[jira] [Updated] (SPARK-14428) [SQL] Allow more flexibility when parsing dates and timestamps in json datasources

2016-04-06 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-14428: - Description: Reading a json with dates and timestamps is limited to predetermined string

[jira] [Updated] (SPARK-14428) [SQL] Allow more flexibility when parsing dates and timestamps in json datasources

2016-04-06 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-14428: - Description: Reading a json with dates and timestamps is limited to predetermined string

[jira] [Created] (SPARK-14428) [SQL] Allow more flexibility when parsing dates and timestamps in json datasources

2016-04-06 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-14428: Summary: [SQL] Allow more flexibility when parsing dates and timestamps in json datasources Key: SPARK-14428 URL: https://issues.apache.org/jira/browse/SPARK-14428

[jira] [Commented] (SPARK-14347) Require Java 8 for Spark 2.x

2016-04-04 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224804#comment-15224804 ] Michel Lemay commented on SPARK-14347: -- Update JavaTypeInference class to support new DateTime types

[jira] [Created] (SPARK-14330) Spark SQL does not infer correct type for joda DateTime

2016-04-01 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-14330: Summary: Spark SQL does not infer correct type for joda DateTime Key: SPARK-14330 URL: https://issues.apache.org/jira/browse/SPARK-14330 Project: Spark

[jira] [Commented] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-31 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220370#comment-15220370 ] Michel Lemay commented on SPARK-12436: -- As if it is commonly used, a quick search on google for

[jira] [Commented] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-31 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220353#comment-15220353 ] Michel Lemay commented on SPARK-12436: -- This example fails to illustrate the issue since the order

[jira] [Comment Edited] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-31 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219966#comment-15219966 ] Michel Lemay edited comment on SPARK-12436 at 3/31/16 2:53 PM: --- Well, right

[jira] [Comment Edited] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-31 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219966#comment-15219966 ] Michel Lemay edited comment on SPARK-12436 at 3/31/16 2:51 PM: --- Well, right

[jira] [Commented] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-31 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219966#comment-15219966 ] Michel Lemay commented on SPARK-12436: -- Well, right now, I don't see what would be the minimum

[jira] [Commented] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-30 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218405#comment-15218405 ] Michel Lemay commented on SPARK-13710: -- Tests passes on Jenkins which is a good thing.. However, my

[jira] [Commented] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-29 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15215871#comment-15215871 ] Michel Lemay commented on SPARK-12436: -- See the comment in 'case VALUE_STRING ..' code piece. //

[jira] [Commented] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-25 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211963#comment-15211963 ] Michel Lemay commented on SPARK-13710: -- I'd gladly do that but the current state of the master

[jira] [Comment Edited] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-25 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211804#comment-15211804 ] Michel Lemay edited comment on SPARK-12436 at 3/25/16 1:48 PM: --- I'm working

[jira] [Commented] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-25 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211804#comment-15211804 ] Michel Lemay commented on SPARK-12436: -- I'm working on a new PR for this issue since the other one

[jira] [Issue Comment Deleted] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-25 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-12436: - Comment: was deleted (was: Is anybody still working on this issue? Seems nothing happened since

[jira] [Commented] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209484#comment-15209484 ] Michel Lemay commented on SPARK-13710: -- Spark coordinates for curator did not change.. It still

[jira] [Commented] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209127#comment-15209127 ] Michel Lemay commented on SPARK-13710: -- Looks like it was renamed to a different package. Therefore

[jira] [Commented] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208949#comment-15208949 ] Michel Lemay commented on SPARK-13710: -- I'm not sure it broke something but seems to be working on

[jira] [Comment Edited] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208849#comment-15208849 ] Michel Lemay edited comment on SPARK-13710 at 3/23/16 5:52 PM: --- Googling

[jira] [Commented] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208849#comment-15208849 ] Michel Lemay commented on SPARK-13710: -- Googling around, it might be a conflict between jline2 and

[jira] [Commented] (SPARK-13710) Spark shell shows ERROR when launching on Windows

2016-03-23 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208529#comment-15208529 ] Michel Lemay commented on SPARK-13710: -- I'm having the same issue. It's worse than that though..

[jira] [Commented] (SPARK-12436) If all values of a JSON field is null, JSON's inferSchema should return NullType instead of StringType

2016-03-21 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204723#comment-15204723 ] Michel Lemay commented on SPARK-12436: -- Is anybody still working on this issue? Seems nothing

[jira] [Commented] (SPARK-13044) saveAsTextFile() doesn't support s3 Signature Version 4

2016-02-26 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169144#comment-15169144 ] Michel Lemay commented on SPARK-13044: -- I updated pom.xml for hadoop-2.7.2 and jets3t 0.9.4 and have

[jira] [Commented] (SPARK-13044) saveAsTextFile() doesn't support s3 Signature Version 4

2016-02-26 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169045#comment-15169045 ] Michel Lemay commented on SPARK-13044: -- For the record, I'm running with hadoop-2.7 profile (which

[jira] [Comment Edited] (SPARK-13044) saveAsTextFile() doesn't support s3 Signature Version 4

2016-02-26 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169013#comment-15169013 ] Michel Lemay edited comment on SPARK-13044 at 2/26/16 1:36 PM: --- I get

[jira] [Comment Edited] (SPARK-13044) saveAsTextFile() doesn't support s3 Signature Version 4

2016-02-26 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169013#comment-15169013 ] Michel Lemay edited comment on SPARK-13044 at 2/26/16 1:34 PM: --- I get

[jira] [Commented] (SPARK-13044) saveAsTextFile() doesn't support s3 Signature Version 4

2016-02-26 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15169013#comment-15169013 ] Michel Lemay commented on SPARK-13044: -- I get something similar. When trying to read KMS encrypted

[jira] [Commented] (SPARK-13493) json to DataFrame to parquet does not respect case sensitiveness

2016-02-25 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15167219#comment-15167219 ] Michel Lemay commented on SPARK-13493: -- On a side note, default value for spark.sql.caseSensitive is

[jira] [Updated] (SPARK-13493) json to DataFrame to parquet does not respect case sensitiveness

2016-02-25 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay updated SPARK-13493: - Description: Not sure where the problem should be fixed exactly but here it is: {noformat} $

[jira] [Created] (SPARK-13493) json to DataFrame to parquet does not respect case sensitiveness

2016-02-25 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-13493: Summary: json to DataFrame to parquet does not respect case sensitiveness Key: SPARK-13493 URL: https://issues.apache.org/jira/browse/SPARK-13493 Project: Spark

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2016-01-28 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121792#comment-15121792 ] Michel Lemay commented on SPARK-10528: -- Your issue is about deleting temp files at time of shutdown.

[jira] [Comment Edited] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2016-01-28 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121493#comment-15121493 ] Michel Lemay edited comment on SPARK-10528 at 1/28/16 1:54 PM: --- This is not

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2016-01-28 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121493#comment-15121493 ] Michel Lemay commented on SPARK-10528: -- This is not the same problem.. Whet you are facing right

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2016-01-07 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15087840#comment-15087840 ] Michel Lemay commented on SPARK-10528: -- Still present on Spark 1.6.0 I was all-in to test new

[jira] [Commented] (SPARK-10528) spark-shell throws java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

2015-11-09 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14996667#comment-14996667 ] Michel Lemay commented on SPARK-10528: -- Looks like the problem is still there with precompiled

[jira] [Closed] (SPARK-9826) Cannot use custom classes in log4j.properties

2015-08-14 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michel Lemay closed SPARK-9826. --- Verified working with a local build of branch-1.4 (1.4.2-snapshot) Cannot use custom classes in

[jira] [Created] (SPARK-9886) Validate usages of Runtime.getRuntime.addShutdownHook

2015-08-12 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-9886: --- Summary: Validate usages of Runtime.getRuntime.addShutdownHook Key: SPARK-9886 URL: https://issues.apache.org/jira/browse/SPARK-9886 Project: Spark Issue

[jira] [Created] (SPARK-9826) Cannot use custom classes in log4j.properties

2015-08-11 Thread Michel Lemay (JIRA)
Michel Lemay created SPARK-9826: --- Summary: Cannot use custom classes in log4j.properties Key: SPARK-9826 URL: https://issues.apache.org/jira/browse/SPARK-9826 Project: Spark Issue Type: Bug