[jira] [Commented] (SPARK-19186) Hash symbol in middle of Sybase database table name causes Spark Exception

2017-01-12 Thread Adrian Schulewitz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821383#comment-15821383 ] Adrian Schulewitz commented on SPARK-19186: --- Hi, I tried enclosing (i) the table name in the

[jira] [Commented] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist

2017-01-12 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821269#comment-15821269 ] roncenzhao commented on SPARK-19187: I think this problem has been resolved in SPARK-17599. I will

[jira] [Commented] (SPARK-18667) input_file_name function does not work with UDF

2017-01-12 Thread Ben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821188#comment-15821188 ] Ben commented on SPARK-18667: - OK, I just tried it with a json and it worked. To be honest I was trying it

[jira] [Comment Edited] (SPARK-18667) input_file_name function does not work with UDF

2017-01-12 Thread Ben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821188#comment-15821188 ] Ben edited comment on SPARK-18667 at 1/12/17 3:01 PM: -- OK, I just tried it with a

[jira] [Commented] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821220#comment-15821220 ] Sean Owen commented on SPARK-19187: --- What's the use case for ignoring this? part of the data that

[jira] [Commented] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist

2017-01-12 Thread roncenzhao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821196#comment-15821196 ] roncenzhao commented on SPARK-19187: In the method `HadoopTableReader.makeRDDForPartitionedTable()`

[jira] [Resolved] (SPARK-19123) KeyProviderException when reading Azure Blobs from Apache Spark

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19123. --- Resolution: Not A Problem > KeyProviderException when reading Azure Blobs from Apache Spark >

[jira] [Commented] (SPARK-18667) input_file_name function does not work with UDF

2017-01-12 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821135#comment-15821135 ] Liang-Chi Hsieh commented on SPARK-18667: - Hi Ben, I've just tried the example codes in current

[jira] [Comment Edited] (SPARK-18667) input_file_name function does not work with UDF

2017-01-12 Thread Ben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821073#comment-15821073 ] Ben edited comment on SPARK-18667 at 1/12/17 2:28 PM: -- I still have the same problem

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

2017-01-12 Thread Sunil Rangwani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821106#comment-15821106 ] Sunil Rangwani commented on SPARK-17463: Hi [~zsxwing] My recordKey that I add to

[jira] [Comment Edited] (SPARK-18667) input_file_name function does not work with UDF

2017-01-12 Thread Ben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821073#comment-15821073 ] Ben edited comment on SPARK-18667 at 1/12/17 2:12 PM: -- I still have the same problem

[jira] [Comment Edited] (SPARK-18667) input_file_name function does not work with UDF

2017-01-12 Thread Ben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821073#comment-15821073 ] Ben edited comment on SPARK-18667 at 1/12/17 2:10 PM: -- I still have the same problem

[jira] [Commented] (SPARK-18667) input_file_name function does not work with UDF

2017-01-12 Thread Ben (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821073#comment-15821073 ] Ben commented on SPARK-18667: - I still have the same problem on pySpark 2.1.0 and Python 3.5.2 with the exact

[jira] [Commented] (SPARK-13857) Feature parity for ALS ML with MLLIB

2017-01-12 Thread Alan Budd (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15821070#comment-15821070 ] Alan Budd commented on SPARK-13857: --- I just had a short email conversation with [~mlnick] with regards

[jira] [Resolved] (SPARK-19055) SparkSession initialization will be associated with invalid SparkContext when new SparkContext is created to replace stopped SparkContext

2017-01-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-19055. - > SparkSession initialization will be associated with invalid SparkContext when > new SparkContext

[jira] [Updated] (SPARK-19035) rand() function in case when cause failed

2017-01-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19035: Target Version/s: 2.0.3, 2.1.1, 2.2.0 > rand() function in case when cause failed >

[jira] [Updated] (SPARK-19035) rand() function in case when cause failed

2017-01-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19035: Description: *In this case:* select case when a=1 then 1

[jira] [Assigned] (SPARK-19063) Add parameter for storage levels to LDA

2017-01-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19063: Assignee: (was: Apache Spark) > Add parameter for storage levels to LDA >

[jira] [Resolved] (SPARK-18969) PullOutNondeterministic should work for Aggregate operator

2017-01-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18969. - Resolution: Fixed Assignee: Wenchen Fan (was: Reynold Xin) Fix

[jira] [Commented] (SPARK-19063) Add parameter for storage levels to LDA

2017-01-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820888#comment-15820888 ] Apache Spark commented on SPARK-19063: -- User 'zdh2292390' has created a pull request for this issue:

[jira] [Assigned] (SPARK-19063) Add parameter for storage levels to LDA

2017-01-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19063: Assignee: Apache Spark > Add parameter for storage levels to LDA >

[jira] [Commented] (SPARK-12076) countDistinct behaves inconsistently

2017-01-12 Thread Paul Zaczkieiwcz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820884#comment-15820884 ] Paul Zaczkieiwcz commented on SPARK-12076: -- Sounds good to me. I wish I could reproduce, but I

[jira] [Closed] (SPARK-19125) Streaming Duration by Count

2017-01-12 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Cândido closed SPARK-19125. - > Streaming Duration by Count > --- > > Key: SPARK-19125 >

[jira] [Resolved] (SPARK-19125) Streaming Duration by Count

2017-01-12 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Cândido resolved SPARK-19125. --- Resolution: Workaround > Streaming Duration by Count > --- > >

[jira] [Issue Comment Deleted] (SPARK-19125) Streaming Duration by Count

2017-01-12 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Cândido updated SPARK-19125: -- Comment: was deleted (was: Hi Mr. Owen, Thank you for your attention. Your alternative

[jira] [Commented] (SPARK-19125) Streaming Duration by Count

2017-01-12 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820780#comment-15820780 ] Paulo Cândido commented on SPARK-19125: --- Hi Mr. Owen, Thank you for your attention. Your

[jira] [Commented] (SPARK-19125) Streaming Duration by Count

2017-01-12 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-19125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820782#comment-15820782 ] Paulo Cândido commented on SPARK-19125: --- Hi Mr. Owen, Thank you for your attention. Your

[jira] [Issue Comment Deleted] (SPARK-13857) Feature parity for ALS ML with MLLIB

2017-01-12 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Pentreath updated SPARK-13857: --- Comment: was deleted (was: My view is in practice brute-force is never going to be efficient

[jira] [Commented] (SPARK-13857) Feature parity for ALS ML with MLLIB

2017-01-12 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820747#comment-15820747 ] Nick Pentreath commented on SPARK-13857: My view is in practice brute-force is never going to be

[jira] [Commented] (SPARK-13857) Feature parity for ALS ML with MLLIB

2017-01-12 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820746#comment-15820746 ] Nick Pentreath commented on SPARK-13857: My view is in practice brute-force is never going to be

[jira] [Updated] (SPARK-18857) SparkSQL ThriftServer hangs while extracting huge data volumes in incremental collect mode

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-18857: -- Fix Version/s: 2.1.1 2.0.3 > SparkSQL ThriftServer hangs while extracting huge data

[jira] [Commented] (SPARK-18209) More robust view canonicalization without full SQL expansion

2017-01-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820681#comment-15820681 ] Apache Spark commented on SPARK-18209: -- User 'jiangxb1987' has created a pull request for this

[jira] [Resolved] (SPARK-19036) Merging dealyed micro batches

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19036. --- Resolution: Won't Fix > Merging dealyed micro batches > - > >

[jira] [Resolved] (SPARK-19045) irrelevant warning when creating a checkpoint dir

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19045. --- > irrelevant warning when creating a checkpoint dir > -

[jira] [Updated] (SPARK-19186) Hash symbol in middle of Sybase database table name causes Spark Exception

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19186: -- Priority: Minor (was: Major) Does quoting the table name do the trick? > Hash symbol in middle of

[jira] [Resolved] (SPARK-19188) Run spark in scala as script file, note not just REPL

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19188. --- Questions belong on u...@spark.apache.org > Run spark in scala as script file, note not just REPL >

[jira] [Resolved] (SPARK-15407) Floor division

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15407. --- Resolution: Not A Problem > Floor division > -- > > Key: SPARK-15407 >

[jira] [Commented] (SPARK-19187) querying from parquet partitioned table throws FileNotFoundException when some partitions' hdfs locations do not exist

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820632#comment-15820632 ] Sean Owen commented on SPARK-19187: --- That sounds like correct behavior, right? > querying from

[jira] [Commented] (SPARK-19156) Example in the doc not working

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820625#comment-15820625 ] Sean Owen commented on SPARK-19156: --- It is clearer. There are examples of lambdas elsewhere in the

[jira] [Commented] (SPARK-19035) rand() function in case when cause failed

2017-01-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820626#comment-15820626 ] Wenchen Fan commented on SPARK-19035: - This is a valid bug. According to the discussion in

[jira] [Commented] (SPARK-19156) Example in the doc not working

2017-01-12 Thread Rafael Guglielmetti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820622#comment-15820622 ] Rafael Guglielmetti commented on SPARK-19156: - And providing both examples? I think the one

[jira] [Resolved] (SPARK-13671) Use different physical plan for existing RDD and data sources

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13671. --- Looks like this was actually resolved by https://github.com/apache/spark/pull/11514 > Use different

[jira] [Reopened] (SPARK-19035) rand() function in case when cause failed

2017-01-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reopened SPARK-19035: - > rand() function in case when cause failed > - > >

[jira] [Resolved] (SPARK-14901) java exception when showing join

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14901. --- Resolution: Not A Problem This looks like an error from Netezza libraries. > java exception when

[jira] [Resolved] (SPARK-19156) Example in the doc not working

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19156. --- Resolution: Fixed Fix Version/s: 2.2.0 > Example in the doc not working >

[jira] [Updated] (SPARK-19156) Example in the doc not working

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19156: -- Assignee: Rafael Guglielmetti > Example in the doc not working > -- > >

[jira] [Commented] (SPARK-19156) Example in the doc not working

2017-01-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820603#comment-15820603 ] Sean Owen commented on SPARK-19156: --- Right now, Spark doesn't require Java 8, though it works with it.

[jira] [Assigned] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19179: Assignee: (was: Apache Spark) > spark.yarn.access.namenodes description is wrong >

[jira] [Assigned] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19179: Assignee: Apache Spark > spark.yarn.access.namenodes description is wrong >

[jira] [Commented] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820566#comment-15820566 ] Apache Spark commented on SPARK-19179: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Updated] (SPARK-19164) Remove unused UserDefinedFunction._broadcast

2017-01-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-19164: Assignee: Maciej Szymkiewicz > Remove unused UserDefinedFunction._broadcast >

[jira] [Resolved] (SPARK-19158) ml.R example fails in yarn-cluster mode due to lacks of e1071 package

2017-01-12 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-19158. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > ml.R example fails in

[jira] [Assigned] (SPARK-19158) ml.R example fails in yarn-cluster mode due to lacks of e1071 package

2017-01-12 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang reassigned SPARK-19158: --- Assignee: Yanbo Liang > ml.R example fails in yarn-cluster mode due to lacks of e1071

[jira] [Updated] (SPARK-19164) Remove unused UserDefinedFunction._broadcast

2017-01-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-19164: Summary: Remove unused UserDefinedFunction._broadcast (was: Review of

[jira] [Resolved] (SPARK-19164) Remove unused UserDefinedFunction._broadcast

2017-01-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-19164. - > Remove unused UserDefinedFunction._broadcast > > >

[jira] [Commented] (SPARK-19164) Review of UserDefinedFunction._broadcast

2017-01-12 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820529#comment-15820529 ] Maciej Szymkiewicz commented on SPARK-19164: [~rxin] I am particularly interested in

[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos

2017-01-12 Thread Jorge Lopez-Malla (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820530#comment-15820530 ] Jorge Lopez-Malla commented on SPARK-16742: --- In Stratio we have had a very busy end of the year

[jira] [Commented] (SPARK-19156) Example in the doc not working

2017-01-12 Thread Rafael Guglielmetti (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15820501#comment-15820501 ] Rafael Guglielmetti commented on SPARK-19156: - One last question: would it be nice to rewrite