[jira] [Updated] (SPARK-5277) SparkSqlSerializer does not register user specified KryoRegistrators

2015-03-27 Thread Max Seiden (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Seiden updated SPARK-5277: -- Affects Version/s: (was: 1.2.0) 1.2.1 1.3.0 > SparkSql

[jira] [Issue Comment Deleted] (SPARK-6577) SparseMatrix should be supported in PySpark

2015-03-27 Thread Manoj Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Kumar updated SPARK-6577: --- Comment: was deleted (was: Can this is be assigned to me in that case? This blocks https://issues.apa

[jira] [Commented] (SPARK-6577) SparseMatrix should be supported in PySpark

2015-03-27 Thread Manoj Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385145#comment-14385145 ] Manoj Kumar commented on SPARK-6577: Can this is be assigned to me in that case? This

[jira] [Commented] (SPARK-6577) SparseMatrix should be supported in PySpark

2015-03-27 Thread Manoj Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385144#comment-14385144 ] Manoj Kumar commented on SPARK-6577: Can this is be assigned to me in that case? This

[jira] [Resolved] (SPARK-6572) When I build Spark 1.3 sbt gives me to following error : unresolved dependency: org.apache.kafka#kafka_2.11;0.8.1.1: not found org.scalamacros#quasiquotes_2.11;2.0.1:

2015-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6572. -- Resolution: Cannot Reproduce Hm, that shouldn't matter either. I have Scala 2.11 installed and is the d

[jira] [Commented] (SPARK-6579) save as parquet with overwrite failed

2015-03-27 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385141#comment-14385141 ] Davies Liu commented on SPARK-6579: --- Hadoop 1.0.4, here is parquet: {code} lib_managed/

[jira] [Created] (SPARK-6581) Metadata is missing when saving parquet file using hadoop 1.0.4

2015-03-27 Thread Pei-Lun Lee (JIRA)
Pei-Lun Lee created SPARK-6581: -- Summary: Metadata is missing when saving parquet file using hadoop 1.0.4 Key: SPARK-6581 URL: https://issues.apache.org/jira/browse/SPARK-6581 Project: Spark Is

[jira] [Assigned] (SPARK-6207) YARN secure cluster mode doesn't obtain a hive-metastore token

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6207: --- Assignee: Apache Spark > YARN secure cluster mode doesn't obtain a hive-metastore token > --

[jira] [Assigned] (SPARK-6207) YARN secure cluster mode doesn't obtain a hive-metastore token

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6207: --- Assignee: (was: Apache Spark) > YARN secure cluster mode doesn't obtain a hive-metastore

[jira] [Commented] (SPARK-6579) save as parquet with overwrite failed

2015-03-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385046#comment-14385046 ] Cheng Lian commented on SPARK-6579: --- Hm, I couldn't reproduce when compiled against Had

[jira] [Commented] (SPARK-4590) Early investigation of parameter server

2015-03-27 Thread Reza Zadeh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385032#comment-14385032 ] Reza Zadeh commented on SPARK-4590: --- The umbrella JIRA for IndexedRDD is at SPARK-2365

[jira] [Commented] (SPARK-4590) Early investigation of parameter server

2015-03-27 Thread Peng Cheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385029#comment-14385029 ] Peng Cheng commented on SPARK-4590: --- Hi Reza, that's great news. Can someone point us to

[jira] [Resolved] (SPARK-6538) Add missing nullable Metastore fields when merging a Parquet schema

2015-03-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian resolved SPARK-6538. --- Resolution: Fixed Fix Version/s: 1.4.0 1.3.1 Issue resolved by pull request

[jira] [Updated] (SPARK-6578) Outbound channel in network library is not thread-safe, can lead to fetch failures

2015-03-27 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-6578: --- Assignee: Marcelo Vanzin > Outbound channel in network library is not thread-safe, can lead to fetch

[jira] [Commented] (SPARK-5894) Add PolynomialMapper

2015-03-27 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384935#comment-14384935 ] Xusen Yin commented on SPARK-5894: -- How about the name "PolynomialExpansion"? > Add Poly

[jira] [Commented] (SPARK-5894) Add PolynomialMapper

2015-03-27 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384913#comment-14384913 ] Xusen Yin commented on SPARK-5894: -- I want to write the polynomial expansion. Pls assign

[jira] [Commented] (SPARK-6479) Create off-heap block storage API (internal)

2015-03-27 Thread Zhan Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384872#comment-14384872 ] Zhan Zhang commented on SPARK-6479: --- I have a short version for this API and will post i

[jira] [Assigned] (SPARK-6466) Remove unnecessary attributes when resolving GroupingSets

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6466: --- Assignee: Apache Spark > Remove unnecessary attributes when resolving GroupingSets >

[jira] [Assigned] (SPARK-6466) Remove unnecessary attributes when resolving GroupingSets

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6466: --- Assignee: (was: Apache Spark) > Remove unnecessary attributes when resolving GroupingSets

[jira] [Created] (SPARK-6580) Optimize LogisticRegressionModel.predictPoint

2015-03-27 Thread Joseph K. Bradley (JIRA)
Joseph K. Bradley created SPARK-6580: Summary: Optimize LogisticRegressionModel.predictPoint Key: SPARK-6580 URL: https://issues.apache.org/jira/browse/SPARK-6580 Project: Spark Issue Typ

[jira] [Updated] (SPARK-2709) Add a tool for certifying Spark API compatiblity

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2709: --- Target Version/s: (was: 1.2.0) > Add a tool for certifying Spark API compatiblity >

[jira] [Reopened] (SPARK-2709) Add a tool for certifying Spark API compatiblity

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell reopened SPARK-2709: This came up in some recent conversations. I actually don't think we ever merged this into Spar

[jira] [Updated] (SPARK-2709) Add a tool for certifying Spark API compatiblity

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-2709: --- Priority: Critical (was: Major) > Add a tool for certifying Spark API compatiblity >

[jira] [Resolved] (SPARK-1844) Support maven-style dependency resolution in sbt build

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-1844. Resolution: Won't Fix Closing given the combination of (a) this is not that important and (b

[jira] [Resolved] (SPARK-4073) Parquet+Snappy can cause significant off-heap memory usage

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-4073. Resolution: Won't Fix I have never seen someone else run into this, so closing as not urgent

[jira] [Resolved] (SPARK-5025) Write a guide for creating well-formed packages for Spark

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell resolved SPARK-5025. Resolution: Won't Fix I'm closing this as wont fix. There are now a bunch of community packa

[jira] [Updated] (SPARK-6255) Python MLlib API missing items: Classification

2015-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-6255: - Description: This JIRA lists items missing in the Python API for this sub-package of MLlib

[jira] [Resolved] (SPARK-6564) SQLContext.emptyDataFrame should contain 0 rows, not 1 row

2015-03-27 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-6564. Resolution: Fixed Fix Version/s: 1.4.0 1.3.1 > SQLContext.emptyDataFrame s

[jira] [Commented] (SPARK-5885) Add VectorAssembler

2015-03-27 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384701#comment-14384701 ] Xiangrui Meng commented on SPARK-5885: -- Yes, we plan to add more wrappers in Python.

[jira] [Assigned] (SPARK-5931) Use consistent naming for time properties

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-5931: --- Assignee: Andrew Or (was: Apache Spark) > Use consistent naming for time properties > --

[jira] [Commented] (SPARK-5931) Use consistent naming for time properties

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384690#comment-14384690 ] Apache Spark commented on SPARK-5931: - User 'ilganeli' has created a pull request for

[jira] [Assigned] (SPARK-5931) Use consistent naming for time properties

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-5931: --- Assignee: Apache Spark (was: Andrew Or) > Use consistent naming for time properties > --

[jira] [Commented] (SPARK-6572) When I build Spark 1.3 sbt gives me to following error : unresolved dependency: org.apache.kafka#kafka_2.11;0.8.1.1: not found org.scalamacros#quasiquotes_2.11;2.0.1

2015-03-27 Thread Frank Domoney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384680#comment-14384680 ] Frank Domoney commented on SPARK-6572: -- Cracked it. I had a scala-2.11 installed a

[jira] [Updated] (SPARK-6571) MatrixFactorizationModel created by load fails on predictAll

2015-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-6571: - Assignee: Xiangrui Meng > MatrixFactorizationModel created by load fails on predictAll > -

[jira] [Created] (SPARK-6579) save as parquet with overwrite failed

2015-03-27 Thread Davies Liu (JIRA)
Davies Liu created SPARK-6579: - Summary: save as parquet with overwrite failed Key: SPARK-6579 URL: https://issues.apache.org/jira/browse/SPARK-6579 Project: Spark Issue Type: Bug Compo

[jira] [Resolved] (SPARK-6526) Add Normalizer transformer

2015-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-6526. -- Resolution: Fixed Issue resolved by pull request 5181 [https://github.com/apache/spark/p

[jira] [Updated] (SPARK-6571) MatrixFactorizationModel created by load fails on predictAll

2015-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-6571: - Target Version/s: 1.3.1, 1.4.0 > MatrixFactorizationModel created by load fails on predict

[jira] [Commented] (SPARK-6479) Create off-heap block storage API (internal)

2015-03-27 Thread Henry Saputra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384525#comment-14384525 ] Henry Saputra commented on SPARK-6479: -- @Steve: Ah cool, thanks for clarifying =) >

[jira] [Commented] (SPARK-6479) Create off-heap block storage API (internal)

2015-03-27 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384520#comment-14384520 ] Steve Loughran commented on SPARK-6479: --- Henry: utterly unrelated. I was merely offe

[jira] [Commented] (SPARK-6577) SparseMatrix should be supported in PySpark

2015-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384517#comment-14384517 ] Joseph K. Bradley commented on SPARK-6577: -- I agree we should add this unless we

[jira] [Assigned] (SPARK-6578) Outbound channel in network library is not thread-safe, can lead to fetch failures

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6578: --- Assignee: (was: Apache Spark) > Outbound channel in network library is not thread-safe, c

[jira] [Commented] (SPARK-6571) MatrixFactorizationModel created by load fails on predictAll

2015-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384491#comment-14384491 ] Joseph K. Bradley commented on SPARK-6571: -- Thanks for the detailed report! I wa

[jira] [Commented] (SPARK-6578) Outbound channel in network library is not thread-safe, can lead to fetch failures

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384493#comment-14384493 ] Apache Spark commented on SPARK-6578: - User 'vanzin' has created a pull request for th

[jira] [Assigned] (SPARK-6578) Outbound channel in network library is not thread-safe, can lead to fetch failures

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6578: --- Assignee: Apache Spark > Outbound channel in network library is not thread-safe, can lead to

[jira] [Created] (SPARK-6578) Outbound channel in network library is not thread-safe, can lead to fetch failures

2015-03-27 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-6578: - Summary: Outbound channel in network library is not thread-safe, can lead to fetch failures Key: SPARK-6578 URL: https://issues.apache.org/jira/browse/SPARK-6578 Pr

[jira] [Assigned] (SPARK-4069) [SPARK-YARN] ApplicationMaster should release all executors' containers before unregistering itself from Yarn RM

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-4069: --- Assignee: (was: Apache Spark) > [SPARK-YARN] ApplicationMaster should release all executo

[jira] [Assigned] (SPARK-4069) [SPARK-YARN] ApplicationMaster should release all executors' containers before unregistering itself from Yarn RM

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-4069: --- Assignee: Apache Spark > [SPARK-YARN] ApplicationMaster should release all executors' contain

[jira] [Commented] (SPARK-4069) [SPARK-YARN] ApplicationMaster should release all executors' containers before unregistering itself from Yarn RM

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384480#comment-14384480 ] Apache Spark commented on SPARK-4069: - User 'PraveenSeluka' has created a pull request

[jira] [Closed] (SPARK-6233) Should spark.ml Models be distributed by default?

2015-03-27 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley closed SPARK-6233. Resolution: Not a Problem Assignee: Joseph K. Bradley I'm closing this discussion beca

[jira] [Commented] (SPARK-6544) Problem with Avro and Kryo Serialization

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384459#comment-14384459 ] Patrick Wendell commented on SPARK-6544: Back-ported to 1.3.1 per discussion on is

[jira] [Updated] (SPARK-6544) Problem with Avro and Kryo Serialization

2015-03-27 Thread Patrick Wendell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Wendell updated SPARK-6544: --- Fix Version/s: 1.3.1 > Problem with Avro and Kryo Serialization >

[jira] [Commented] (SPARK-6577) SparseMatrix should be supported in PySpark

2015-03-27 Thread Manoj Kumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384440#comment-14384440 ] Manoj Kumar commented on SPARK-6577: [~mengxr] [~josephkb] Could you please confirm th

[jira] [Created] (SPARK-6577) SparseMatrix should be supported in PySpark

2015-03-27 Thread Manoj Kumar (JIRA)
Manoj Kumar created SPARK-6577: -- Summary: SparseMatrix should be supported in PySpark Key: SPARK-6577 URL: https://issues.apache.org/jira/browse/SPARK-6577 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-6576) DenseMatrix in PySpark should support indexing

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384432#comment-14384432 ] Apache Spark commented on SPARK-6576: - User 'MechCoder' has created a pull request for

[jira] [Assigned] (SPARK-6576) DenseMatrix in PySpark should support indexing

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6576: --- Assignee: (was: Apache Spark) > DenseMatrix in PySpark should support indexing >

[jira] [Assigned] (SPARK-6576) DenseMatrix in PySpark should support indexing

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6576: --- Assignee: Apache Spark > DenseMatrix in PySpark should support indexing > ---

[jira] [Updated] (SPARK-6575) Add configuration to disable schema merging while converting metastore Parquet tables

2015-03-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-6575: -- Description: Consider a metastore Parquet table that # doesn't have schema evolution issue # has lots of

[jira] [Updated] (SPARK-6575) Add configuration to disable schema merging while converting metastore Parquet tables

2015-03-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-6575: -- Description: Consider a metastore Parquet table that # doesn't have schema evolution issue # has lots of

[jira] [Created] (SPARK-6576) DenseMatrix in PySpark should support indexing

2015-03-27 Thread Manoj Kumar (JIRA)
Manoj Kumar created SPARK-6576: -- Summary: DenseMatrix in PySpark should support indexing Key: SPARK-6576 URL: https://issues.apache.org/jira/browse/SPARK-6576 Project: Spark Issue Type: New Feat

[jira] [Commented] (SPARK-6489) Optimize lateral view with explode to not read unnecessary columns

2015-03-27 Thread Adnan Khan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384401#comment-14384401 ] Adnan Khan commented on SPARK-6489: --- [~dreamquster]: have you started working on this?

[jira] [Commented] (SPARK-6565) Deprecate jsonRDD and replace it by jsonDataFrame / jsonDF

2015-03-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384386#comment-14384386 ] Cheng Lian commented on SPARK-6565: --- Ah, makes sense. Thanks! > Deprecate jsonRDD and r

[jira] [Assigned] (SPARK-6575) Add configuration to disable schema merging while converting metastore Parquet tables

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6575: --- Assignee: Apache Spark (was: Cheng Lian) > Add configuration to disable schema merging while

[jira] [Commented] (SPARK-6575) Add configuration to disable schema merging while converting metastore Parquet tables

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384378#comment-14384378 ] Apache Spark commented on SPARK-6575: - User 'liancheng' has created a pull request for

[jira] [Assigned] (SPARK-6575) Add configuration to disable schema merging while converting metastore Parquet tables

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6575: --- Assignee: Cheng Lian (was: Apache Spark) > Add configuration to disable schema merging while

[jira] [Created] (SPARK-6575) Add configuration to disable schema merging while converting metastore Parquet tables

2015-03-27 Thread Cheng Lian (JIRA)
Cheng Lian created SPARK-6575: - Summary: Add configuration to disable schema merging while converting metastore Parquet tables Key: SPARK-6575 URL: https://issues.apache.org/jira/browse/SPARK-6575 Project

[jira] [Closed] (SPARK-6168) Expose some of the collection classes as DeveloperApi

2015-03-27 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan closed SPARK-6168. -- Resolution: Won't Fix > Expose some of the collection classes as DeveloperApi >

[jira] [Resolved] (SPARK-6574) Python Example sql.py not working in version 1.3

2015-03-27 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-6574. - Resolution: Fixed Fix Version/s: 1.4.0 1.3.1 Issue resolved by p

[jira] [Commented] (SPARK-6168) Expose some of the collection classes as DeveloperApi

2015-03-27 Thread Mridul Muralidharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384347#comment-14384347 ] Mridul Muralidharan commented on SPARK-6168: Closing based on internal discuss

[jira] [Resolved] (SPARK-6550) Add PreAnalyzer to keep logical plan consistent across DataFrame

2015-03-27 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-6550. - Resolution: Fixed Fix Version/s: 1.4.0 1.3.1 Issue resolved by p

[jira] [Commented] (SPARK-6479) Create off-heap block storage API (internal)

2015-03-27 Thread Henry Saputra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384304#comment-14384304 ] Henry Saputra commented on SPARK-6479: -- [~ste...@apache.org], could you clarify more

[jira] [Resolved] (SPARK-6565) Deprecate jsonRDD and replace it by jsonDataFrame / jsonDF

2015-03-27 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-6565. - Resolution: Won't Fix > Deprecate jsonRDD and replace it by jsonDataFrame / jsonDF > -

[jira] [Updated] (SPARK-6564) SQLContext.emptyDataFrame should contain 0 rows, not 1 row

2015-03-27 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-6564: --- Priority: Blocker (was: Major) > SQLContext.emptyDataFrame should contain 0 rows, not 1 row > ---

[jira] [Commented] (SPARK-5885) Add VectorAssembler

2015-03-27 Thread Omede Firouz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384240#comment-14384240 ] Omede Firouz commented on SPARK-5885: - [~mengxr], do you have plans for adding python

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Robert Kanter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384235#comment-14384235 ] Robert Kanter commented on SPARK-5493: -- Correct. It turns out that Oozie doesn't act

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Brock Noland (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384218#comment-14384218 ] Brock Noland commented on SPARK-5493: - I don't know 100% about how oozie works but I b

[jira] [Commented] (SPARK-4727) Add "dimensional" RDDs (time series, spatial)

2015-03-27 Thread Simon Ouellette (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384213#comment-14384213 ] Simon Ouellette commented on SPARK-4727: This is a great idea, I need a TimeSeries

[jira] [Updated] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-5493: - Description: When using kerberos, services may want to use spark-submit to submit jobs as a separa

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384211#comment-14384211 ] Thomas Graves commented on SPARK-5493: -- Basically oozie is the one that does the prox

[jira] [Assigned] (SPARK-6574) Python Example sql.py not working in version 1.3

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6574: --- Assignee: Apache Spark (was: Davies Liu) > Python Example sql.py not working in version 1.3

[jira] [Assigned] (SPARK-6574) Python Example sql.py not working in version 1.3

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6574: --- Assignee: Davies Liu (was: Apache Spark) > Python Example sql.py not working in version 1.3

[jira] [Commented] (SPARK-6574) Python Example sql.py not working in version 1.3

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384172#comment-14384172 ] Apache Spark commented on SPARK-6574: - User 'davies' has created a pull request for th

[jira] [Commented] (SPARK-6565) Deprecate jsonRDD and replace it by jsonDataFrame / jsonDF

2015-03-27 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384161#comment-14384161 ] Michael Armbrust commented on SPARK-6565: - It is not that it returns an RDD, it is

[jira] [Created] (SPARK-6574) Python Example sql.py not working in version 1.3

2015-03-27 Thread Davies Liu (JIRA)
Davies Liu created SPARK-6574: - Summary: Python Example sql.py not working in version 1.3 Key: SPARK-6574 URL: https://issues.apache.org/jira/browse/SPARK-6574 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384145#comment-14384145 ] Marcelo Vanzin commented on SPARK-5493: --- Direct link: https://github.com/apache/hive

[jira] [Commented] (SPARK-4660) JavaSerializer uses wrong classloader

2015-03-27 Thread sam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384142#comment-14384142 ] sam commented on SPARK-4660: Furthermore it seems this issue is more likely to happen when I t

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384140#comment-14384140 ] Marcelo Vanzin commented on SPARK-5493: --- I'm not terribly familiar with how Oozie ha

[jira] [Commented] (SPARK-6572) When I build Spark 1.3 sbt gives me to following error : unresolved dependency: org.apache.kafka#kafka_2.11;0.8.1.1: not found org.scalamacros#quasiquotes_2.11;2.0.1

2015-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384037#comment-14384037 ] Sean Owen commented on SPARK-6572: -- It builds correctly for me in branch 1.3 with {{build

[jira] [Commented] (SPARK-6569) Kafka directInputStream logs what appear to be incorrect warnings

2015-03-27 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384038#comment-14384038 ] Cody Koeninger commented on SPARK-6569: --- I set it as warn because an empty batch can

[jira] [Commented] (SPARK-6572) When I build Spark 1.3 sbt gives me to following error : unresolved dependency: org.apache.kafka#kafka_2.11;0.8.1.1: not found org.scalamacros#quasiquotes_2.11;2.0.1

2015-03-27 Thread Frank Domoney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384006#comment-14384006 ] Frank Domoney commented on SPARK-6572: -- the correct URL for the kafka is kafka_2.11-0

[jira] [Commented] (SPARK-6572) When I build Spark 1.3 sbt gives me to following error : unresolved dependency: org.apache.kafka#kafka_2.11;0.8.1.1: not found org.scalamacros#quasiquotes_2.11;2.0.1

2015-03-27 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383999#comment-14383999 ] Cheng Lian commented on SPARK-6572: --- Would you please provide exact command line you use

[jira] [Updated] (SPARK-6573) expect pandas null values as numpy.nan (not only as None)

2015-03-27 Thread Fabian Boehnlein (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Boehnlein updated SPARK-6573: Issue Type: Sub-task (was: Improvement) Parent: SPARK-6116 > expect pandas null val

[jira] [Commented] (SPARK-6569) Kafka directInputStream logs what appear to be incorrect warnings

2015-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383973#comment-14383973 ] Sean Owen commented on SPARK-6569: -- [~c...@koeninger.org] what do you think of the warnin

[jira] [Created] (SPARK-6573) expect pandas null values as numpy.nan (not only as None)

2015-03-27 Thread Fabian Boehnlein (JIRA)
Fabian Boehnlein created SPARK-6573: --- Summary: expect pandas null values as numpy.nan (not only as None) Key: SPARK-6573 URL: https://issues.apache.org/jira/browse/SPARK-6573 Project: Spark

[jira] [Reopened] (SPARK-6569) Kafka directInputStream logs what appear to be incorrect warnings

2015-03-27 Thread Platon Potapov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Platon Potapov reopened SPARK-6569: --- Sean, please explain if the condition really mandates a warning being logged. The scenario in whi

[jira] [Created] (SPARK-6572) When I build Spark 1.3 sbt gives me to following error : unresolved dependency: org.apache.kafka#kafka_2.11;0.8.1.1: not found org.scalamacros#quasiquotes_2.11;2.0.1:

2015-03-27 Thread Frank Domoney (JIRA)
Frank Domoney created SPARK-6572: Summary: When I build Spark 1.3 sbt gives me to following error : unresolved dependency: org.apache.kafka#kafka_2.11;0.8.1.1: not found org.scalamacros#quasiquotes_2.11;2.0.1: not found [error] Total time: 27

[jira] [Resolved] (SPARK-6544) Problem with Avro and Kryo Serialization

2015-03-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6544. -- Resolution: Fixed Fix Version/s: 1.4.0 Issue resolved by pull request 5193 [https://github.com/ap

[jira] [Comment Edited] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383856#comment-14383856 ] Thomas Graves edited comment on SPARK-5493 at 3/27/15 2:00 PM: -

[jira] [Commented] (SPARK-5493) Support proxy users under kerberos

2015-03-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383856#comment-14383856 ] Thomas Graves commented on SPARK-5493: -- [~vanzin] I must be missing something. Why

[jira] [Assigned] (SPARK-6558) Utils.getCurrentUserName returns the full principal name instead of login name

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-6558: --- Assignee: Thomas Graves (was: Apache Spark) > Utils.getCurrentUserName returns the full prin

[jira] [Commented] (SPARK-6558) Utils.getCurrentUserName returns the full principal name instead of login name

2015-03-27 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383843#comment-14383843 ] Apache Spark commented on SPARK-6558: - User 'tgravescs' has created a pull request for

  1   2   >