[jira] [Commented] (SPARK-21212) Can't use Count(*) with Order Clause

2017-06-26 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063968#comment-16063968 ] Sean Owen commented on SPARK-21212: --- Yes but you are not selecting the thing you order

[jira] [Commented] (SPARK-21222) Move elimination of Distinct clause from analyzer to optimizer

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064441#comment-16064441 ] Sean Owen commented on SPARK-21222: --- Do you mean this is the same as "SELECT MAX(a) FRO

[jira] [Updated] (SPARK-21226) Save empty dataframe in pyspark prints nothing

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21226: -- Priority: Minor (was: Major) What is the error? > Save empty dataframe in pyspark prints nothing > --

[jira] [Commented] (SPARK-21223) Thread-safety issue in FsHistoryProvider

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16064676#comment-16064676 ] Sean Owen commented on SPARK-21223: --- [~gostop_zlx] this overlaps a lot with SPARK-21078

[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065336#comment-16065336 ] Sean Owen commented on SPARK-21215: --- I'm not sure what you're referring to. The user@ l

[jira] [Commented] (SPARK-21230) Spark Encoder with mysql Enum and data truncated Error

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065338#comment-16065338 ] Sean Owen commented on SPARK-21230: --- This does also not look like a useful JIRA. It loo

[jira] [Commented] (SPARK-21215) Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065348#comment-16065348 ] Sean Owen commented on SPARK-21215: --- Not sure what you're looking at, but the mailing l

[jira] [Commented] (SPARK-21137) Spark reads many small files slowly

2017-06-27 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065475#comment-16065475 ] Sean Owen commented on SPARK-21137: --- OK, so it is something that could be optimized in

[jira] [Resolved] (SPARK-21234) When the function returns Option[Iterator[_]] is None,then get on None will cause java.util.NoSuchElementException: None.get

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21234. --- Resolution: Invalid Not if it's known the value exists. I don't see you've established any actual pr

[jira] [Commented] (SPARK-21233) Support pluggable offset storage

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066088#comment-16066088 ] Sean Owen commented on SPARK-21233: --- Where would you put it instead? Kafka already prov

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066471#comment-16066471 ] Sean Owen commented on SPARK-21227: --- Yes, I think this is ultimately related to two dif

[jira] [Commented] (SPARK-21244) KMeans applied to processed text day clumps almost all documents into one cluster

2017-06-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066833#comment-16066833 ] Sean Owen commented on SPARK-21244: --- There's no detail here that suggests a Spark bug.

[jira] [Assigned] (SPARK-21210) Javadoc 8 fixes for ML shared param traits

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21210: - Assignee: Nick Pentreath > Javadoc 8 fixes for ML shared param traits >

[jira] [Resolved] (SPARK-21210) Javadoc 8 fixes for ML shared param traits

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21210. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 18420 [https://github.co

[jira] [Assigned] (SPARK-21240) Fix code style for constructing and stopping a SparkContext in UT

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21240: - Assignee: jin xing Issue Type: Improvement (was: Bug) > Fix code style for constructing a

[jira] [Resolved] (SPARK-21240) Fix code style for constructing and stopping a SparkContext in UT

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21240. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18454 [https://github.co

[jira] [Resolved] (SPARK-21135) On history server page,duration of incompleted applications should be hidden instead of showing up as 0

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21135. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18351 [https://github.co

[jira] [Assigned] (SPARK-21135) On history server page,duration of incompleted applications should be hidden instead of showing up as 0

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21135: - Assignee: Jinhua Fu > On history server page,duration of incompleted applications should be hidd

[jira] [Commented] (SPARK-21252) The duration times showed by spark web UI are inaccurate

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068069#comment-16068069 ] Sean Owen commented on SPARK-21252: --- They are rounded, and '3 seconds' is unlikely to b

[jira] [Commented] (SPARK-21252) The duration times showed by spark web UI are inaccurate

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068085#comment-16068085 ] Sean Owen commented on SPARK-21252: --- You can drill into individual stage times, right?

[jira] [Updated] (SPARK-21252) The duration times showed by spark web UI are inaccurate

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21252: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) I don't know what the underlyin

[jira] [Commented] (SPARK-21255) NPE when creating encoder for enum

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16068520#comment-16068520 ] Sean Owen commented on SPARK-21255: --- Is the change to omit declaringClass too? > NPE w

[jira] [Resolved] (SPARK-21254) History UI: Taking over 1 minute for initial page display

2017-06-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21254. --- Resolution: Duplicate Duplicate of lots of JIRAs related to making the initial read faster > History

[jira] [Commented] (SPARK-21263) NumberFormatException is not thrown while converting an invalid string to float/double

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069664#comment-16069664 ] Sean Owen commented on SPARK-21263: --- This is because: {code} java.text.NumberFormat.ge

[jira] [Commented] (SPARK-21255) NPE when creating encoder for enum

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069668#comment-16069668 ] Sean Owen commented on SPARK-21255: --- [~mike0sv] OK, feel free to open a PR to exclude i

[jira] [Resolved] (SPARK-19294) improve LocalLDAModel save/load scaling for large models

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-19294. --- Resolution: Duplicate > improve LocalLDAModel save/load scaling for large models > --

[jira] [Commented] (SPARK-21254) History UI: Taking over 1 minute for initial page display

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069735#comment-16069735 ] Sean Owen commented on SPARK-21254: --- I linked the umbrella above? > History UI: Taking

[jira] [Resolved] (SPARK-21265) Cache method could specified name

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21265. --- Resolution: Duplicate Have a look through JIRA first please > Cache method could specified name > --

[jira] [Resolved] (SPARK-4131) Support "Writing data into the filesystem from queries"

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-4131. -- Resolution: Not A Problem This is long since obsolete. You can easily create a DataFrame from a query,

[jira] [Commented] (SPARK-21263) NumberFormatException is not thrown while converting an invalid string to float/double

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069911#comment-16069911 ] Sean Owen commented on SPARK-21263: --- CC [~falaki] as well for the original code Yeah,

[jira] [Commented] (SPARK-21254) History UI: Taking over 1 minute for initial page display

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069913#comment-16069913 ] Sean Owen commented on SPARK-21254: --- Do none of the sub-tickets overlap with this? CC [

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070070#comment-16070070 ] Sean Owen commented on SPARK-21227: --- You should certainly make your app less sensitive

[jira] [Commented] (SPARK-21227) Unicode in Json field causes AnalysisException when selecting from Dataframe

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070091#comment-16070091 ] Sean Owen commented on SPARK-21227: --- I don't know, I think there is a real issue here s

[jira] [Commented] (SPARK-21268) Redundant collectAsMap in KMeans

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070114#comment-16070114 ] Sean Owen commented on SPARK-21268: --- I don't think it's redundant, because totalContrib

[jira] [Commented] (SPARK-21268) Redundant collectAsMap in KMeans

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070217#comment-16070217 ] Sean Owen commented on SPARK-21268: --- You can just update this issue. > Redundant colle

[jira] [Resolved] (SPARK-21156) Spark cannot handle multiple KMS server configuration

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21156. --- Resolution: Not A Problem Closing as not a problem particular to Spark > Spark cannot handle multipl

[jira] [Commented] (SPARK-21270) Improvement for memory config.

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070336#comment-16070336 ] Sean Owen commented on SPARK-21270: --- How would Spark know how much of the total memory

[jira] [Resolved] (SPARK-21223) Thread-safety issue in FsHistoryProvider

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21223. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18430 [https://github.co

[jira] [Assigned] (SPARK-21223) Thread-safety issue in FsHistoryProvider

2017-06-30 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21223: - Assignee: zenglinxi Priority: Minor (was: Major) > Thread-safety issue in FsHistoryProvider

[jira] [Comment Edited] (SPARK-21206) the window slice of Dstream is wrong

2017-07-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062284#comment-16062284 ] Sean Owen edited comment on SPARK-21206 at 7/1/17 8:40 AM: --- [~s

[jira] [Commented] (SPARK-21206) the window slice of Dstream is wrong

2017-07-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071105#comment-16071105 ] Sean Owen commented on SPARK-21206: --- I'm still not clear what you're saying, even after

[jira] [Resolved] (SPARK-21233) Support pluggable offset storage

2017-07-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21233. --- Resolution: Not A Problem > Support pluggable offset storage > > >

[jira] [Resolved] (SPARK-21244) KMeans applied to processed text day clumps almost all documents into one cluster

2017-07-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21244. --- Resolution: Not A Problem > KMeans applied to processed text day clumps almost all documents into one

[jira] [Resolved] (SPARK-21170) Utils.tryWithSafeFinallyAndFailureCallbacks throws IllegalArgumentException: Self-suppression not permitted

2017-07-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21170. --- Resolution: Fixed Assignee: Devaraj K Fix Version/s: 2.3.0 2.2.1 R

[jira] [Assigned] (SPARK-21260) Remove the unused OutputFakerExec

2017-07-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21260: - Assignee: Jiang Xingbo Priority: Trivial (was: Minor) > Remove the unused OutputFakerExec >

[jira] [Resolved] (SPARK-21260) Remove the unused OutputFakerExec

2017-07-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21260. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18473 [https://github.co

[jira] [Resolved] (SPARK-21206) the window slice of Dstream is wrong

2017-07-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21206. --- Resolution: Invalid I still don't understand what you're trying to point out. The original doc drawi

[jira] [Commented] (SPARK-15526) Shade JPMML

2017-07-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071552#comment-16071552 ] Sean Owen commented on SPARK-15526: --- I do think we should shade in 2.3.0 here. Yes, sha

[jira] [Commented] (SPARK-21206) the window slice of Dstream is wrong

2017-07-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071709#comment-16071709 ] Sean Owen commented on SPARK-21206: --- I think you are interpreting the log message to me

[jira] [Updated] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21280: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) I don't think it is intended to

[jira] [Commented] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-02 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071752#comment-16071752 ] Sean Owen commented on SPARK-21280: --- There are plenty of classes that are public in the

[jira] [Commented] (SPARK-21263) NumberFormatException is not thrown while converting an invalid string to float/double

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072241#comment-16072241 ] Sean Owen commented on SPARK-21263: --- I believe I favor taking the small behavior change

[jira] [Commented] (SPARK-20814) Mesos scheduler does not respect spark.executor.extraClassPath configuration

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072258#comment-16072258 ] Sean Owen commented on SPARK-20814: --- [~laurentcoder] there is no 2.2.0, yet. It is in t

[jira] [Updated] (SPARK-21239) Support WAL recover in windows

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21239: -- Fix Version/s: (was: 2.2.1) (was: 2.1.2) > Support WAL recover in windows >

[jira] [Updated] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21241: -- Fix Version/s: (was: 2.3.0) > Add intercept to StreamingLinearRegressionWithSGD > -

[jira] [Updated] (SPARK-21093) Multiple gapply execution occasionally failed in SparkR

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21093: -- Fix Version/s: (was: 2.3.0) > Multiple gapply execution occasionally failed in SparkR > --

[jira] [Updated] (SPARK-21232) New built-in SQL function - Data_Type

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21232: -- Fix Version/s: (was: 2.2.0) > New built-in SQL function - Data_Type > -

[jira] [Updated] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21241: -- Flags: (was: Patch) Labels: (was: patch) > Add intercept to StreamingLinearRegressionWithSGD >

[jira] [Updated] (SPARK-21241) Add intercept to StreamingLinearRegressionWithSGD

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21241: -- Target Version/s: (was: 2.3.0) > Add intercept to StreamingLinearRegressionWithSGD >

[jira] [Updated] (SPARK-21285) VectorAssembler should report the column name when data type used is not supported

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21285: -- Issue Type: Improvement (was: Bug) Agree, seems like a good improvement to the error. > VectorAssembl

[jira] [Commented] (SPARK-21287) Cannot use Int.MIN_VALUE as Spark SQL fetchsize

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072388#comment-16072388 ] Sean Owen commented on SPARK-21287: --- Yeah, I'm familiar with this special value. Is it

[jira] [Commented] (SPARK-21287) Cannot use Int.MIN_VALUE as Spark SQL fetchsize

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072414#comment-16072414 ] Sean Owen commented on SPARK-21287: --- It's not supposed to do that right -- you're sayin

[jira] [Commented] (SPARK-21287) Cannot use Int.MIN_VALUE as Spark SQL fetchsize

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072506#comment-16072506 ] Sean Owen commented on SPARK-21287: --- I know, but this is what fetch size is supposed to

[jira] [Commented] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072734#comment-16072734 ] Sean Owen commented on SPARK-21280: --- Looking at the class, I don't think it was ever in

[jira] [Commented] (SPARK-21287) Cannot use Int.MIN_VALUE as Spark SQL fetchsize

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072752#comment-16072752 ] Sean Owen commented on SPARK-21287: --- Yeah, maybe a method to validate or even set the f

[jira] [Commented] (SPARK-21288) Several files are missing in the results of the execution of the spark application.

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072763#comment-16072763 ] Sean Owen commented on SPARK-21288: --- I don't think this is a Spark problem. See things

[jira] [Commented] (SPARK-17993) Spark prints an avalanche of warning messages from Parquet when reading parquet files written by older versions of Parquet-mr

2017-07-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073141#comment-16073141 ] Sean Owen commented on SPARK-17993: --- [~jhpoelen] those are not files that this change t

[jira] [Commented] (SPARK-21299) except is throwing the fallowing exception after perform dropDuplicates on the Dataset object

2017-07-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073498#comment-16073498 ] Sean Owen commented on SPARK-21299: --- Just dumping a plan / stacktrace doesn't help. Thi

[jira] [Commented] (SPARK-21302) history server WebUI show HTTP ERROR 500

2017-07-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073518#comment-16073518 ] Sean Owen commented on SPARK-21302: --- How do you reproduce it? are there early errors? t

[jira] [Commented] (SPARK-21303) Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill

2017-07-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073540#comment-16073540 ] Sean Owen commented on SPARK-21303: --- I don't think this narrows it down nearly enough t

[jira] [Commented] (SPARK-21305) The BKM (best known methods) of using native BLAS to improvement ML/MLLIB performance

2017-07-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073789#comment-16073789 ] Sean Owen commented on SPARK-21305: --- OK, can you propose a concrete change to the docs?

[jira] [Resolved] (SPARK-21268) Move center calculations to a distributed map in KMeans

2017-07-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21268. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18491 [https://github.co

[jira] [Assigned] (SPARK-21268) Move center calculations to a distributed map in KMeans

2017-07-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21268: - Assignee: Guillaume Dardelet > Move center calculations to a distributed map in KMeans > ---

[jira] [Commented] (SPARK-21263) NumberFormatException is not thrown while converting an invalid string to float/double

2017-07-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074292#comment-16074292 ] Sean Owen commented on SPARK-21263: --- [~hyukjin.kwon] [~falaki] Yeah I saw "PERMISSIVE"

[jira] [Updated] (SPARK-21314) ByteArrayMethods.arrayEquals could use some optimizations

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21314: -- Issue Type: Improvement (was: Bug) (not a bug) I would be surprised if it made much difference, becau

[jira] [Resolved] (SPARK-20858) Document ListenerBus event queue size property

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20858. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18476 [https://github.co

[jira] [Updated] (SPARK-21318) The exception message thrown by `lookupFunction` is ambiguous.

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21318: -- Issue Type: Improvement (was: Bug) (not a bug) Internally, can it distinguish between something that r

[jira] [Commented] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074928#comment-16074928 ] Sean Owen commented on SPARK-21280: --- Not sure what you mean here. Serialization can mea

[jira] [Commented] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075008#comment-16075008 ] Sean Owen commented on SPARK-21321: --- There is no detail here about what you're suggesti

[jira] [Closed] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-21280. - > org.apache.spark.util.sketch.BloomFilter not bean compliant > -

[jira] [Resolved] (SPARK-21280) org.apache.spark.util.sketch.BloomFilter not bean compliant

2017-07-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21280. --- Resolution: Not A Problem ... and the problem is? In any event I think the original issue posed here

[jira] [Resolved] (SPARK-21325) The shell of 'spark-submit' about '--jars' and '--fils', jars and files can be placed on local and hdfs.

2017-07-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21325. --- Resolution: Invalid I'm preemptively resolving these types of JIRAs, as there is no detail at all he

[jira] [Commented] (SPARK-21331) java.lang.NullPointerException for certain methods in classes of MLlib

2017-07-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077701#comment-16077701 ] Sean Owen commented on SPARK-21331: --- I don't see evidence that the NPE is related to co

[jira] [Assigned] (SPARK-21313) ConsoleSink's string representation

2017-07-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21313: - Assignee: Jacek Laskowski > ConsoleSink's string representation > --

[jira] [Resolved] (SPARK-21313) ConsoleSink's string representation

2017-07-07 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21313. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18539 [https://github.co

[jira] [Commented] (SPARK-21341) Spark 2.1.1: I want to be able to serialize wordVectors on Word2VecModel

2017-07-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079019#comment-16079019 ] Sean Owen commented on SPARK-21341: --- Hm, I don't think you normally save a model with s

[jira] [Resolved] (SPARK-21347) Performance issues with KryoSerializer

2017-07-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21347. --- Resolution: Not A Problem I think this is more of question, and about a change or configuration in Kr

[jira] [Commented] (SPARK-21344) BinaryType comparison does signed byte array comparison

2017-07-08 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079023#comment-16079023 ] Sean Owen commented on SPARK-21344: --- Bytes are signed, as are other int types in Spark

[jira] [Resolved] (SPARK-21352) Memory Usage in Spark Streaming

2017-07-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21352. --- Resolution: Invalid Please point questions to StackOverflow or the mailing list. > Memory Usage in S

[jira] [Resolved] (SPARK-21352) Memory Usage in Spark Streaming

2017-07-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21352. --- Resolution: Fixed That does not imply this is the right place. This is for proposing changes and dia

[jira] [Resolved] (SPARK-21352) Memory Usage in Spark Streaming

2017-07-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21352. --- Resolution: Invalid > Memory Usage in Spark Streaming > --- > >

[jira] [Reopened] (SPARK-21352) Memory Usage in Spark Streaming

2017-07-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-21352: --- > Memory Usage in Spark Streaming > --- > > Key: SPARK-21352

[jira] [Closed] (SPARK-21352) Memory Usage in Spark Streaming

2017-07-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen closed SPARK-21352. - > Memory Usage in Spark Streaming > --- > > Key: SPARK-21352 >

[jira] [Commented] (SPARK-21358) Argument of repartitionandsortwithinpartitions at pyspark

2017-07-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079909#comment-16079909 ] Sean Owen commented on SPARK-21358: --- Good catch, yeah, I'd just remove that final argum

[jira] [Resolved] (SPARK-21303) Web-UI shows some Jobs get stuck randomly and stays like that. Neither able to kill

2017-07-09 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21303. --- Resolution: Not A Problem I would look at thread dumps (available in the UI) to see where tasks are

[jira] [Commented] (SPARK-21359) frequency discretizer

2017-07-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080080#comment-16080080 ] Sean Owen commented on SPARK-21359: --- I can't understand from this code what the functio

[jira] [Resolved] (SPARK-21361) Spark failing to query SQL Server. Query contains a column having space in where clause

2017-07-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21361. --- Resolution: Duplicate > Spark failing to query SQL Server. Query contains a column having space in

[jira] [Updated] (SPARK-21360) Spark failing to query SQL Server. Query contains a column having space in where clause

2017-07-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21360: -- Priority: Major (was: Blocker) [~ferozkhan] please back up and read http://spark.apache.org/contributi

[jira] [Commented] (SPARK-21359) frequency discretizer

2017-07-10 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080339#comment-16080339 ] Sean Owen commented on SPARK-21359: --- No, what you are describing is pretty much the def

[jira] [Resolved] (SPARK-21359) frequency discretizer

2017-07-11 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21359. --- Resolution: Not A Problem The discretization is a little approximate, for performance at scale. Here

<    1   2   3   4   5   6   7   8   9   10   >