[jira] [Comment Edited] (SPARK-15639) Try to push down filter at RowGroups level for parquet reader

2016-08-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415591#comment-15415591 ] Davies Liu edited comment on SPARK-15639 at 8/10/16 5:07 PM: - Merged

[jira] [Commented] (SPARK-16994) Filter and limit are illegally permuted.

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415569#comment-15415569 ] Dongjoon Hyun commented on SPARK-16994: --- Hi, [~TPolzer]. Indeed, it is. `PushDownPredicate` seems

[jira] [Created] (SPARK-16998) select($"column1", explode($"column2")) is extremely slow

2016-08-10 Thread TobiasP (JIRA)
TobiasP created SPARK-16998: --- Summary: select($"column1", explode($"column2")) is extremely slow Key: SPARK-16998 URL: https://issues.apache.org/jira/browse/SPARK-16998 Project: Spark Issue Type:

[jira] [Updated] (SPARK-15899) file scheme should be used correctly

2016-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15899: -- Fix Version/s: (was: 2.0.1) > file scheme should be used correctly >

[jira] [Commented] (SPARK-16929) Speculation-related synchronization bottleneck in checkSpeculatableTasks

2016-08-10 Thread Nicholas Brown (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415490#comment-15415490 ] Nicholas Brown commented on SPARK-16929: One other thing, after looking at the code more closely,

[jira] [Resolved] (SPARK-15639) Try to push down filter at RowGroups level for parquet reader

2016-08-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu resolved SPARK-15639. Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Target

[jira] [Assigned] (SPARK-16994) Filter and limit are illegally permuted.

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16994: Assignee: (was: Apache Spark) > Filter and limit are illegally permuted. >

[jira] [Assigned] (SPARK-16994) Filter and limit are illegally permuted.

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16994: Assignee: Apache Spark > Filter and limit are illegally permuted. >

[jira] [Commented] (SPARK-16994) Filter and limit are illegally permuted.

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415605#comment-15415605 ] Apache Spark commented on SPARK-16994: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-16995) TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr

2016-08-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415265#comment-15415265 ] Cédric Perriard commented on SPARK-16995: - I added it to the description. Should I include the

[jira] [Commented] (SPARK-16951) Alternative implementation of NOT IN to Anti-join

2016-08-10 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415350#comment-15415350 ] Nattavut Sutyanyong commented on SPARK-16951: - On the incorrect result in Case 1, it is good

[jira] [Created] (SPARK-16995) TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr

2016-08-10 Thread JIRA
Cédric Perriard created SPARK-16995: --- Summary: TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr Key: SPARK-16995 URL:

[jira] [Updated] (SPARK-16995) TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr

2016-08-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cédric Perriard updated SPARK-16995: Description: A TreeNodeException is thrown when executing the following minimal example in

[jira] [Resolved] (SPARK-15899) file scheme should be used correctly

2016-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15899. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 Resolved by

[jira] [Created] (SPARK-16996) Hive ACID delta files not seen

2016-08-10 Thread Benjamin BONNET (JIRA)
Benjamin BONNET created SPARK-16996: --- Summary: Hive ACID delta files not seen Key: SPARK-16996 URL: https://issues.apache.org/jira/browse/SPARK-16996 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-12370) Documentation should link to examples from its own release version

2016-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415200#comment-15415200 ] Sean Owen commented on SPARK-12370: --- The docs build does define a variable that indicates the current

[jira] [Commented] (SPARK-16996) Hive ACID delta files not seen

2016-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415279#comment-15415279 ] Sean Owen commented on SPARK-16996: --- I am not sure if this is supported, given the version of Hive and

[jira] [Commented] (SPARK-16997) Allow loading of JSON float values as TimestampType

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415373#comment-15415373 ] Apache Spark commented on SPARK-16997: -- User 'lalinsky' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16997) Allow loading of JSON float values as TimestampType

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16997: Assignee: Apache Spark > Allow loading of JSON float values as TimestampType >

[jira] [Closed] (SPARK-16949) Add lower/uppper bounds for AUC calculation

2016-08-10 Thread Patrick Baier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Baier closed SPARK-16949. - Resolution: Won't Fix > Add lower/uppper bounds for AUC calculation >

[jira] [Commented] (SPARK-16951) Alternative implementation of NOT IN to Anti-join

2016-08-10 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415340#comment-15415340 ] Nattavut Sutyanyong commented on SPARK-16951: - I think I will make this the last comment on

[jira] [Commented] (SPARK-12844) Spark documentation should be more precise about the algebraic properties of functions in various transformations

2016-08-10 Thread Jagadeesan A S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415194#comment-15415194 ] Jagadeesan A S commented on SPARK-12844: Started working on this. > Spark documentation should

[jira] [Commented] (SPARK-16949) Add lower/uppper bounds for AUC calculation

2016-08-10 Thread Patrick Baier (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415329#comment-15415329 ] Patrick Baier commented on SPARK-16949: --- Okay, maybe this should be really solved in user-code

[jira] [Commented] (SPARK-15882) Discuss distributed linear algebra in spark.ml package

2016-08-10 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415345#comment-15415345 ] Yanbo Liang commented on SPARK-15882: - I think we should make this migration finished before the next

[jira] [Commented] (SPARK-16995) TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr

2016-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415243#comment-15415243 ] Sean Owen commented on SPARK-16995: --- You haven't shown the exception -- this is typically useful. >

[jira] [Comment Edited] (SPARK-16951) Alternative implementation of NOT IN to Anti-join

2016-08-10 Thread Nattavut Sutyanyong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415340#comment-15415340 ] Nattavut Sutyanyong edited comment on SPARK-16951 at 8/10/16 2:23 PM:

[jira] [Assigned] (SPARK-16997) Allow loading of JSON float values as TimestampType

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16997: Assignee: (was: Apache Spark) > Allow loading of JSON float values as TimestampType >

[jira] [Commented] (SPARK-16993) model.transform without label column in random forest regression

2016-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415222#comment-15415222 ] Sean Owen commented on SPARK-16993: --- You need a label for training and evaluation. You do not need one

[jira] [Updated] (SPARK-16995) TreeNodeException when flat mapping RelationalGroupedDataset created from DataFrame containing a column created with lit/expr

2016-08-10 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cédric Perriard updated SPARK-16995: Description: A TreeNodeException is thrown when executing the following minimal example in

[jira] [Updated] (SPARK-16989) fileSystem.getFileStatus throw exception in EventLoggingListener

2016-08-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16989: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) If the user specifies a log

[jira] [Created] (SPARK-16997) Allow loading of JSON float values as TimestampType

2016-08-10 Thread Lukas Lalinsky (JIRA)
Lukas Lalinsky created SPARK-16997: -- Summary: Allow loading of JSON float values as TimestampType Key: SPARK-16997 URL: https://issues.apache.org/jira/browse/SPARK-16997 Project: Spark

[jira] [Commented] (SPARK-16989) fileSystem.getFileStatus throw exception in EventLoggingListener

2016-08-10 Thread Hong Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416320#comment-15416320 ] Hong Shen commented on SPARK-16989: --- Thanks for your comment. In our cluster, the log directory is a

[jira] [Created] (SPARK-17007) Move test data files into a test-data folder

2016-08-10 Thread Peter Lee (JIRA)
Peter Lee created SPARK-17007: - Summary: Move test data files into a test-data folder Key: SPARK-17007 URL: https://issues.apache.org/jira/browse/SPARK-17007 Project: Spark Issue Type: Sub-task

[jira] [Assigned] (SPARK-17007) Move test data files into a test-data folder

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17007: Assignee: Apache Spark > Move test data files into a test-data folder >

[jira] [Created] (SPARK-17005) Fix warning "method tpe in trait AnnotationApi is deprecated"

2016-08-10 Thread Xin Ren (JIRA)
Xin Ren created SPARK-17005: --- Summary: Fix warning "method tpe in trait AnnotationApi is deprecated" Key: SPARK-17005 URL: https://issues.apache.org/jira/browse/SPARK-17005 Project: Spark Issue

[jira] [Assigned] (SPARK-17005) Fix warning "method tpe in trait AnnotationApi is deprecated"

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17005: Assignee: (was: Apache Spark) > Fix warning "method tpe in trait AnnotationApi is

[jira] [Commented] (SPARK-17005) Fix warning "method tpe in trait AnnotationApi is deprecated"

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416262#comment-15416262 ] Apache Spark commented on SPARK-17005: -- User 'keypointt' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17005) Fix warning "method tpe in trait AnnotationApi is deprecated"

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17005: Assignee: Apache Spark > Fix warning "method tpe in trait AnnotationApi is deprecated" >

[jira] [Created] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2016-08-10 Thread Hamel Ajay Kothari (JIRA)
Hamel Ajay Kothari created SPARK-17006: -- Summary: WithColumn Performance Degrades with Number of Invocations Key: SPARK-17006 URL: https://issues.apache.org/jira/browse/SPARK-17006 Project: Spark

[jira] [Updated] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2016-08-10 Thread Hamel Ajay Kothari (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hamel Ajay Kothari updated SPARK-17006: --- Description: Consider the following test case. We create a dataframe with 100

[jira] [Closed] (SPARK-17004) Fix warning "method declarations in class TypeApi is deprecated"

2016-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-17004. --- Resolution: Won't Fix This will break Scala 2.10. > Fix warning "method declarations in class

[jira] [Commented] (SPARK-17007) Move test data files into a test-data folder

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416314#comment-15416314 ] Apache Spark commented on SPARK-17007: -- User 'petermaxlee' has created a pull request for this

[jira] [Assigned] (SPARK-17007) Move test data files into a test-data folder

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17007: Assignee: (was: Apache Spark) > Move test data files into a test-data folder >

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415823#comment-15415823 ] Reynold Xin commented on SPARK-16975: - cc [~dongjoon] do you have time to look into this? >

[jira] [Resolved] (SPARK-16710) SparkR spark.glm should support weightCol

2016-08-10 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivaram Venkataraman resolved SPARK-16710. --- Resolution: Fixed Fix Version/s: 2.1.0 Issue resolved by pull request

[jira] [Assigned] (SPARK-16999) Convert Keyword `path` of Data Source Tables to Lower Case

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16999: Assignee: (was: Apache Spark) > Convert Keyword `path` of Data Source Tables to Lower

[jira] [Commented] (SPARK-16999) Convert Keyword `path` of Data Source Tables to Lower Case

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415689#comment-15415689 ] Apache Spark commented on SPARK-16999: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Assigned] (SPARK-16999) Convert Keyword `path` of Data Source Tables to Lower Case

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16999: Assignee: Apache Spark > Convert Keyword `path` of Data Source Tables to Lower Case >

[jira] [Created] (SPARK-17000) Spark cannot connect to secure metastore when using custom metastore jars

2016-08-10 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-17000: -- Summary: Spark cannot connect to secure metastore when using custom metastore jars Key: SPARK-17000 URL: https://issues.apache.org/jira/browse/SPARK-17000

[jira] [Commented] (SPARK-14887) Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits

2016-08-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415710#comment-15415710 ] Davies Liu commented on SPARK-14887: Do you have a large CaseWhen in this query? > Generated

[jira] [Resolved] (SPARK-17000) Spark cannot connect to secure metastore when using custom metastore jars

2016-08-10 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-17000. Resolution: Duplicate My bad, duplicate. > Spark cannot connect to secure metastore when

[jira] [Commented] (SPARK-16802) joins.LongToUnsafeRowMap crashes with ArrayIndexOutOfBoundsException

2016-08-10 Thread Miao Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415698#comment-15415698 ] Miao Wang commented on SPARK-16802: --- Sorry for missing your comments. My email box doesn't show up the

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415932#comment-15415932 ] Dongjoon Hyun commented on SPARK-16975: --- I made a sample case having similar behaviors. I think

[jira] [Comment Edited] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415941#comment-15415941 ] Dongjoon Hyun edited comment on SPARK-16975 at 8/10/16 8:18 PM: Ah,

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415903#comment-15415903 ] Dongjoon Hyun commented on SPARK-16975: --- Hi, [~immerrr]. I can not reproduce your situation, but

[jira] [Commented] (SPARK-16227) Json schema inference fails when `:` exists in file path

2016-08-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415991#comment-15415991 ] Davies Liu commented on SPARK-16227: [~brkyvz] I can't reproduce this in master (2.1-snapshot),

[jira] [Comment Edited] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415903#comment-15415903 ] Dongjoon Hyun edited comment on SPARK-16975 at 8/10/16 7:57 PM: Hi,

[jira] [Created] (SPARK-17001) Enable standardScaler to standardize sparse vectors when withMean=True

2016-08-10 Thread Tobi Bosede (JIRA)
Tobi Bosede created SPARK-17001: --- Summary: Enable standardScaler to standardize sparse vectors when withMean=True Key: SPARK-17001 URL: https://issues.apache.org/jira/browse/SPARK-17001 Project: Spark

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415957#comment-15415957 ] Dongjoon Hyun commented on SPARK-16975: --- Let me dig more. I can find more general solution for this

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread immerrr again (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415961#comment-15415961 ] immerrr again commented on SPARK-16975: --- oh, that's unfortunate. coming from python world,

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415941#comment-15415941 ] Dongjoon Hyun commented on SPARK-16975: --- Ah, [~rxin]. For this issue, we should add a migration

[jira] [Assigned] (SPARK-14887) Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits

2016-08-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu reassigned SPARK-14887: -- Assignee: Davies Liu > Generated SpecificUnsafeProjection Exceeds JVM Code Size Limits >

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415966#comment-15415966 ] Dongjoon Hyun commented on SPARK-16975: --- In the python, could you give the command string to write

[jira] [Closed] (SPARK-11638) Run Spark on Mesos with bridge networking

2016-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-11638. --- Resolution: Later Marking this as later since the original pull request has been closed. > Run

[jira] [Created] (SPARK-16987) Add spark-default.conf property to define https port for spark history server

2016-08-10 Thread Yesha Vora (JIRA)
Yesha Vora created SPARK-16987: -- Summary: Add spark-default.conf property to define https port for spark history server Key: SPARK-16987 URL: https://issues.apache.org/jira/browse/SPARK-16987 Project:

[jira] [Commented] (SPARK-16921) RDD/DataFrame persist() and cache() should return Python context managers

2016-08-10 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414819#comment-15414819 ] Nick Pentreath commented on SPARK-16921: I've taken a look at it and have something close to

[jira] [Created] (SPARK-16988) spark history server log needs to be fixed to show https url when ssl is enabled

2016-08-10 Thread Yesha Vora (JIRA)
Yesha Vora created SPARK-16988: -- Summary: spark history server log needs to be fixed to show https url when ssl is enabled Key: SPARK-16988 URL: https://issues.apache.org/jira/browse/SPARK-16988

[jira] [Updated] (SPARK-16989) fileSystem.getFileStatus throw exception in EventLoggingListener

2016-08-10 Thread Hong Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Shen updated SPARK-16989: -- Description: If log directory does not exist, it will throw exception as the follow log. {code}

[jira] [Updated] (SPARK-6744) Add support for CROSS JOIN syntax

2016-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-6744: --- Assignee: Herman van Hovell > Add support for CROSS JOIN syntax > - >

[jira] [Resolved] (SPARK-6744) Add support for CROSS JOIN syntax

2016-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-6744. Resolution: Fixed Fix Version/s: 2.0.0 This has been fixed in 2.0.0. > Add support for

[jira] [Commented] (SPARK-16966) App Name is a randomUUID even when "spark.app.name" exists

2016-08-10 Thread Weiqing Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414778#comment-15414778 ] Weiqing Yang commented on SPARK-16966: -- Thanks for the feedback. I will update my PR to remove those

[jira] [Created] (SPARK-16989) fileSystem.getFileStatus throw exception in EventLoggingListener

2016-08-10 Thread Hong Shen (JIRA)
Hong Shen created SPARK-16989: - Summary: fileSystem.getFileStatus throw exception in EventLoggingListener Key: SPARK-16989 URL: https://issues.apache.org/jira/browse/SPARK-16989 Project: Spark

[jira] [Created] (SPARK-16990) Define the data structure to hold the statistics for CBO

2016-08-10 Thread Ron Hu (JIRA)
Ron Hu created SPARK-16990: -- Summary: Define the data structure to hold the statistics for CBO Key: SPARK-16990 URL: https://issues.apache.org/jira/browse/SPARK-16990 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16227) Json schema inference fails when `:` exists in file path

2016-08-10 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415994#comment-15415994 ] Davies Liu commented on SPARK-16227: Can reproduce this by change `jsont` to `jsont:1` > Json schema

[jira] [Assigned] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16975: Assignee: Apache Spark > Spark-2.0.0 unable to infer schema for parquet data written by

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread immerrr again (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416013#comment-15416013 ] immerrr again commented on SPARK-16975: --- Yes, a seemingly similar one. > Spark-2.0.0 unable to

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416038#comment-15416038 ] Dongjoon Hyun commented on SPARK-16975: --- Thank you. See you tomorrow! :) > Spark-2.0.0 unable to

[jira] [Comment Edited] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread immerrr again (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415998#comment-15415998 ] immerrr again edited comment on SPARK-16975 at 8/10/16 8:57 PM: You mean

[jira] [Commented] (SPARK-16975) Spark-2.0.0 unable to infer schema for parquet data written by Spark-1.6.2

2016-08-10 Thread immerrr again (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415998#comment-15415998 ] immerrr again commented on SPARK-16975: --- You mean the one I use to write the data, I don't have the

[jira] [Created] (SPARK-17009) Use a new SparkSession for each test case

2016-08-10 Thread Peter Lee (JIRA)
Peter Lee created SPARK-17009: - Summary: Use a new SparkSession for each test case Key: SPARK-17009 URL: https://issues.apache.org/jira/browse/SPARK-17009 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2016-08-10 Thread Hamel Ajay Kothari (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416463#comment-15416463 ] Hamel Ajay Kothari commented on SPARK-17006: The unrolled code doesn't really make a

[jira] [Commented] (SPARK-12370) Documentation should link to examples from its own release version

2016-08-10 Thread Jagadeesan A S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416518#comment-15416518 ] Jagadeesan A S commented on SPARK-12370: So, there is not need for PR..? or can i raise PR for

[jira] [Commented] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-08-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416538#comment-15416538 ] Yin Huai commented on SPARK-16670: -- My bad. I meant to use {{(part = 2 OR part = 3) AND value < 100}}

[jira] [Resolved] (SPARK-16670) Partition pruning doesn't work when queries contain disjunctions involving partition columns

2016-08-10 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-16670. -- Resolution: Not A Problem > Partition pruning doesn't work when queries contain disjunctions involving

[jira] [Updated] (SPARK-8045) Stack overflow in query parser when there is too many where

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-8045: - Attachment: longSql.txt > Stack overflow in query parser when there is too many where >

[jira] [Commented] (SPARK-17010) [MINOR]Wrong description in memory management document

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416423#comment-15416423 ] Apache Spark commented on SPARK-17010: -- User 'WangTaoTheTonic' has created a pull request for this

[jira] [Resolved] (SPARK-8045) Stack overflow in query parser when there is too many where

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-8045. -- Resolution: Fixed Fix Version/s: 2.0.0 New parser can handle this. > Stack overflow in

[jira] [Assigned] (SPARK-17010) [MINOR]Wrong description in memory management document

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17010: Assignee: (was: Apache Spark) > [MINOR]Wrong description in memory management

[jira] [Commented] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416453#comment-15416453 ] Dongjoon Hyun commented on SPARK-17006: --- Hi, [~hkothari]. It's interesting experiment. By the way,

[jira] [Comment Edited] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416453#comment-15416453 ] Dongjoon Hyun edited comment on SPARK-17006 at 8/11/16 3:15 AM: Hi,

[jira] [Resolved] (SPARK-17008) Normalize query results using sorting

2016-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-17008. - Resolution: Fixed Assignee: Peter Lee Fix Version/s: 2.1.0 > Normalize query

[jira] [Created] (SPARK-17008) Normalize query results using sorting

2016-08-10 Thread Peter Lee (JIRA)
Peter Lee created SPARK-17008: - Summary: Normalize query results using sorting Key: SPARK-17008 URL: https://issues.apache.org/jira/browse/SPARK-17008 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-17009) Use a new SparkSession for each test case

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416387#comment-15416387 ] Apache Spark commented on SPARK-17009: -- User 'petermaxlee' has created a pull request for this

[jira] [Assigned] (SPARK-17008) Normalize query results using sorting

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17008: Assignee: Apache Spark > Normalize query results using sorting >

[jira] [Assigned] (SPARK-17009) Use a new SparkSession for each test case

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17009: Assignee: (was: Apache Spark) > Use a new SparkSession for each test case >

[jira] [Assigned] (SPARK-17008) Normalize query results using sorting

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17008: Assignee: (was: Apache Spark) > Normalize query results using sorting >

[jira] [Commented] (SPARK-17008) Normalize query results using sorting

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416386#comment-15416386 ] Apache Spark commented on SPARK-17008: -- User 'petermaxlee' has created a pull request for this

[jira] [Assigned] (SPARK-17009) Use a new SparkSession for each test case

2016-08-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17009: Assignee: Apache Spark > Use a new SparkSession for each test case >

[jira] [Closed] (SPARK-17005) Fix warning "method tpe in trait AnnotationApi is deprecated"

2016-08-10 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-17005. --- Resolution: Won't Fix This will break Scala 2.10. > Fix warning "method tpe in trait AnnotationApi

[jira] [Commented] (SPARK-17006) WithColumn Performance Degrades with Number of Invocations

2016-08-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15416466#comment-15416466 ] Dongjoon Hyun commented on SPARK-17006: --- I see. I got your point. Thank you, [~hkothari] >

  1   2   3   >