[jira] [Created] (SPARK-21440) Refactor ArrowConverters and add DecimalType, ArrayType and StructType support.

2017-07-17 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-21440: - Summary: Refactor ArrowConverters and add DecimalType, ArrayType and StructType support. Key: SPARK-21440 URL: https://issues.apache.org/jira/browse/SPARK-21440 Pro

[jira] [Resolved] (SPARK-21327) ArrayConstructor should handle an array of typecode 'l' as long rather than int in Python 2.

2017-07-06 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-21327. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18553 [https://g

[jira] [Assigned] (SPARK-21327) ArrayConstructor should handle an array of typecode 'l' as long rather than int in Python 2.

2017-07-06 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-21327: - Assignee: Takuya Ueshin > ArrayConstructor should handle an array of typecode 'l' as lon

[jira] [Created] (SPARK-21327) ArrayConstructor should handle an array of typecode 'l' as long rather than int in Python 2.

2017-07-06 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-21327: - Summary: ArrayConstructor should handle an array of typecode 'l' as long rather than int in Python 2. Key: SPARK-21327 URL: https://issues.apache.org/jira/browse/SPARK-21327

[jira] [Created] (SPARK-21300) ExternalMapToCatalyst should null-check map key prior to converting to internal value.

2017-07-04 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-21300: - Summary: ExternalMapToCatalyst should null-check map key prior to converting to internal value. Key: SPARK-21300 URL: https://issues.apache.org/jira/browse/SPARK-21300

[jira] [Resolved] (SPARK-21264) Omitting columns with 'how' specified in join in PySpark throws NPE

2017-07-03 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-21264. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18484 [https://g

[jira] [Assigned] (SPARK-21264) Omitting columns with 'how' specified in join in PySpark throws NPE

2017-07-03 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-21264: - Assignee: Hyukjin Kwon > Omitting columns with 'how' specified in join in PySpark throws

[jira] [Resolved] (SPARK-20830) PySpark wrappers for explode_outer and posexplode_outer

2017-06-21 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-20830. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18049 [https://g

[jira] [Assigned] (SPARK-20830) PySpark wrappers for explode_outer and posexplode_outer

2017-06-21 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-20830: - Assignee: Maciej Szymkiewicz > PySpark wrappers for explode_outer and posexplode_outer >

[jira] [Assigned] (SPARK-21125) PySpark context missing function to set Job Description.

2017-06-21 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-21125: - Assignee: Shane Jarvie > PySpark context missing function to set Job Description. >

[jira] [Resolved] (SPARK-21125) PySpark context missing function to set Job Description.

2017-06-21 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-21125. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18332 [https://g

[jira] [Resolved] (SPARK-17914) Spark SQL casting to TimestampType with nanosecond results in incorrect timestamp

2017-06-12 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-17914. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 Issue resolved by pull

[jira] [Assigned] (SPARK-17914) Spark SQL casting to TimestampType with nanosecond results in incorrect timestamp

2017-06-12 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-17914: - Assignee: Anton Okolnychyi > Spark SQL casting to TimestampType with nanosecond results

[jira] [Assigned] (SPARK-19732) DataFrame.fillna() does not work for bools in PySpark

2017-06-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-19732: - Assignee: Ruben Berenguel > DataFrame.fillna() does not work for bools in PySpark >

[jira] [Resolved] (SPARK-19732) DataFrame.fillna() does not work for bools in PySpark

2017-06-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-19732. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18164 [https://g

[jira] [Resolved] (SPARK-20910) Build-in SQL Function Support - UUID

2017-06-01 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-20910. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18136 [https://g

[jira] [Assigned] (SPARK-20910) Build-in SQL Function Support - UUID

2017-06-01 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin reassigned SPARK-20910: - Assignee: Yuming Wang > Build-in SQL Function Support - UUID > -

[jira] [Resolved] (SPARK-20909) Build-in SQL Function Support - DAYOFWEEK

2017-05-29 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-20909. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18134 [https://g

[jira] [Created] (SPARK-20668) Modify ScalaUDF to handle nullability.

2017-05-08 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-20668: - Summary: Modify ScalaUDF to handle nullability. Key: SPARK-20668 URL: https://issues.apache.org/jira/browse/SPARK-20668 Project: Spark Issue Type: Improvem

[jira] [Commented] (SPARK-20588) from_utc_timestamp causes bottleneck

2017-05-07 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000290#comment-16000290 ] Takuya Ueshin commented on SPARK-20588: --- I agree with caching per thread for now.

[jira] [Commented] (SPARK-12297) Add work-around for Parquet/Hive int96 timestamp bug.

2017-05-07 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16000213#comment-16000213 ] Takuya Ueshin commented on SPARK-12297: --- Issue resolved by pull request 16781 https

[jira] [Resolved] (SPARK-12297) Add work-around for Parquet/Hive int96 timestamp bug.

2017-05-07 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-12297. --- Resolution: Fixed Fix Version/s: 2.3.0 > Add work-around for Parquet/Hive int96 timest

[jira] [Resolved] (SPARK-10364) Support Parquet logical type TIMESTAMP_MILLIS

2017-04-03 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-10364. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 15332 [https://g

[jira] [Created] (SPARK-20201) Flaky Test: org.apache.spark.sql.catalyst.expressions.OrderingSuite

2017-04-03 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-20201: - Summary: Flaky Test: org.apache.spark.sql.catalyst.expressions.OrderingSuite Key: SPARK-20201 URL: https://issues.apache.org/jira/browse/SPARK-20201 Project: Spark

[jira] [Created] (SPARK-20200) Flaky Test: org.apache.spark.rdd.LocalCheckpointSuite

2017-04-03 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-20200: - Summary: Flaky Test: org.apache.spark.rdd.LocalCheckpointSuite Key: SPARK-20200 URL: https://issues.apache.org/jira/browse/SPARK-20200 Project: Spark Issue

[jira] [Created] (SPARK-20131) Flaky Test: org.apache.spark.streaming.StreamingContextSuite

2017-03-28 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-20131: - Summary: Flaky Test: org.apache.spark.streaming.StreamingContextSuite Key: SPARK-20131 URL: https://issues.apache.org/jira/browse/SPARK-20131 Project: Spark

[jira] [Commented] (SPARK-18939) Timezone support in partition values.

2016-12-19 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763375#comment-15763375 ] Takuya Ueshin commented on SPARK-18939: --- Yes, I think so. > Timezone support in pa

[jira] [Created] (SPARK-18939) Timezone support in partition values.

2016-12-19 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18939: - Summary: Timezone support in partition values. Key: SPARK-18939 URL: https://issues.apache.org/jira/browse/SPARK-18939 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-18623) Add `returnNullable` to `StaticInvoke` and modify it to handle properly.

2016-11-29 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18623: - Summary: Add `returnNullable` to `StaticInvoke` and modify it to handle properly. Key: SPARK-18623 URL: https://issues.apache.org/jira/browse/SPARK-18623 Project: S

[jira] [Created] (SPARK-18585) Use `ev.isNull = "false"` if possible for Janino to have a chance to optimize.

2016-11-24 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18585: - Summary: Use `ev.isNull = "false"` if possible for Janino to have a chance to optimize. Key: SPARK-18585 URL: https://issues.apache.org/jira/browse/SPARK-18585 Proj

[jira] [Created] (SPARK-18583) Fix nullability of InputFileName.

2016-11-24 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18583: - Summary: Fix nullability of InputFileName. Key: SPARK-18583 URL: https://issues.apache.org/jira/browse/SPARK-18583 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-18500) Make GenericStrategy be able to prune plans by itself after placeholders are replaced.

2016-11-17 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18500: - Summary: Make GenericStrategy be able to prune plans by itself after placeholders are replaced. Key: SPARK-18500 URL: https://issues.apache.org/jira/browse/SPARK-18500

[jira] [Created] (SPARK-18467) Refactor StaticInvoke, Invoke and NewInstance.

2016-11-15 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18467: - Summary: Refactor StaticInvoke, Invoke and NewInstance. Key: SPARK-18467 URL: https://issues.apache.org/jira/browse/SPARK-18467 Project: Spark Issue Type:

[jira] [Created] (SPARK-18442) Fix nullability of WrapOption.

2016-11-14 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18442: - Summary: Fix nullability of WrapOption. Key: SPARK-18442 URL: https://issues.apache.org/jira/browse/SPARK-18442 Project: Spark Issue Type: Bug Co

[jira] [Created] (SPARK-18398) Fix nullabilities of MapObjects and optimize not to check null if lambda is not nullable.

2016-11-10 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-18398: - Summary: Fix nullabilities of MapObjects and optimize not to check null if lambda is not nullable. Key: SPARK-18398 URL: https://issues.apache.org/jira/browse/SPARK-18398

[jira] [Created] (SPARK-17985) Bump commons-lang3 version to 3.5.

2016-10-17 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-17985: - Summary: Bump commons-lang3 version to 3.5. Key: SPARK-17985 URL: https://issues.apache.org/jira/browse/SPARK-17985 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-17703) Add unnamed version of addReferenceObj for minor objects.

2016-09-28 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-17703: -- Description: There are many `minor` objects in references, which are extracted to the generate

[jira] [Created] (SPARK-17703) Add unnamed version of addReferenceObj for minor objects.

2016-09-28 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-17703: - Summary: Add unnamed version of addReferenceObj for minor objects. Key: SPARK-17703 URL: https://issues.apache.org/jira/browse/SPARK-17703 Project: Spark I

[jira] [Created] (SPARK-17702) Code generation including too many mutable states exceeds JVM size limit.

2016-09-27 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-17702: - Summary: Code generation including too many mutable states exceeds JVM size limit. Key: SPARK-17702 URL: https://issues.apache.org/jira/browse/SPARK-17702 Project:

[jira] [Created] (SPARK-16189) Add ExistingRDD logical plan for input with RDD to have a chance to eliminate serialize/deserialize.

2016-06-24 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-16189: - Summary: Add ExistingRDD logical plan for input with RDD to have a chance to eliminate serialize/deserialize. Key: SPARK-16189 URL: https://issues.apache.org/jira/browse/SPARK-1

[jira] [Closed] (SPARK-15980) Add PushPredicateThroughObjectConsumer rule to Optimizer.

2016-06-24 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin closed SPARK-15980. - Resolution: Duplicate > Add PushPredicateThroughObjectConsumer rule to Optimizer. > -

[jira] [Created] (SPARK-16167) RowEncoder should preserve array/map type nullability.

2016-06-23 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-16167: - Summary: RowEncoder should preserve array/map type nullability. Key: SPARK-16167 URL: https://issues.apache.org/jira/browse/SPARK-16167 Project: Spark Issu

[jira] [Created] (SPARK-15980) Add PushPredicateThroughObjectConsumer rule to Optimizer.

2016-06-15 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15980: - Summary: Add PushPredicateThroughObjectConsumer rule to Optimizer. Key: SPARK-15980 URL: https://issues.apache.org/jira/browse/SPARK-15980 Project: Spark I

[jira] [Created] (SPARK-15915) CacheManager should use canonicalized plan for planToCache.

2016-06-13 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15915: - Summary: CacheManager should use canonicalized plan for planToCache. Key: SPARK-15915 URL: https://issues.apache.org/jira/browse/SPARK-15915 Project: Spark

[jira] [Created] (SPARK-15870) DataFrame can't execute after uncacheTable.

2016-06-10 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15870: - Summary: DataFrame can't execute after uncacheTable. Key: SPARK-15870 URL: https://issues.apache.org/jira/browse/SPARK-15870 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-14752) LazilyGenerateOrdering throws NullPointerException

2016-06-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312215#comment-15312215 ] Takuya Ueshin commented on SPARK-14752: --- I agree that we should avoid null checks o

[jira] [Updated] (SPARK-15483) IncrementalExecution should use extra strategies.

2016-05-23 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-15483: -- Description: Extra strategies does not work for streams because {{IncrementalExecution}} uses m

[jira] [Created] (SPARK-15483) IncrementalExecution should use extra strategies.

2016-05-23 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15483: - Summary: IncrementalExecution should use extra strategies. Key: SPARK-15483 URL: https://issues.apache.org/jira/browse/SPARK-15483 Project: Spark Issue Typ

[jira] [Created] (SPARK-15400) CreateNamedStruct and CreateNamedStructUnsafe should preserve metadata of value expressions if it is NamedExpression.

2016-05-19 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15400: - Summary: CreateNamedStruct and CreateNamedStructUnsafe should preserve metadata of value expressions if it is NamedExpression. Key: SPARK-15400 URL: https://issues.apache.org/ji

[jira] [Created] (SPARK-15313) EmbedSerializerInFilter rule should keep exprIds of output of surrounded SerializeFromObject.

2016-05-13 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15313: - Summary: EmbedSerializerInFilter rule should keep exprIds of output of surrounded SerializeFromObject. Key: SPARK-15313 URL: https://issues.apache.org/jira/browse/SPARK-15313

[jira] [Updated] (SPARK-15308) RowEncoder should preserve nested column name.

2016-05-12 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-15308: -- Description: The following code generates wrong schema: {code} val schema = new StructType().a

[jira] [Created] (SPARK-15308) RowEncoder should preserve nested column name.

2016-05-12 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-15308: - Summary: RowEncoder should preserve nested column name. Key: SPARK-15308 URL: https://issues.apache.org/jira/browse/SPARK-15308 Project: Spark Issue Type:

[jira] [Updated] (SPARK-13902) Make DAGScheduler not to create duplicate stage.

2016-05-08 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-13902: -- Description: {{DAGScheduler}} sometimes generate incorrect stage graph. Suppose you have the f

[jira] [Updated] (SPARK-13902) Make DAGScheduler not to create duplicate stage.

2016-05-08 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-13902: -- Description: {{DAGScheduler}} sometimes generate incorrect stage graph. Suppose you have the f

[jira] [Commented] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-04-26 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257721#comment-15257721 ] Takuya Ueshin commented on SPARK-13902: --- I found that the example above does not re

[jira] [Commented] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-04-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257496#comment-15257496 ] Takuya Ueshin commented on SPARK-13902: --- Yes, exactly. > Make DAGScheduler.getAnce

[jira] [Updated] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-04-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-13902: -- Description: {{DAGScheduler}} sometimes generate incorrect stage graph. Some stages are generat

[jira] [Commented] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-04-25 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257481#comment-15257481 ] Takuya Ueshin commented on SPARK-13902: --- I'm sorry, I made a mistake. I should have

[jira] [Updated] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-04-24 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-13902: -- Description: {{DAGScheduler}} sometimes generate incorrect stage graph. Some stages are generat

[jira] [Updated] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-04-24 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-13902: -- Description: {{DAGScheduler}} sometimes generate incorrect stage graph. Some stages are generat

[jira] [Updated] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-04-24 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-13902: -- Description: {{DAGScheduler}} sometimes generate incorrect stage graph. Some stages are generat

[jira] [Created] (SPARK-14793) Code generation for large complex type exceeds JVM size limit.

2016-04-20 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-14793: - Summary: Code generation for large complex type exceeds JVM size limit. Key: SPARK-14793 URL: https://issues.apache.org/jira/browse/SPARK-14793 Project: Spark

[jira] [Created] (SPARK-14269) Eliminate unnecessary submitStage() call.

2016-03-30 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-14269: - Summary: Eliminate unnecessary submitStage() call. Key: SPARK-14269 URL: https://issues.apache.org/jira/browse/SPARK-14269 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-03-15 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-13902: -- Description: {{DAGScheduler}} sometimes generate incorrect stage graph. Some stages are generat

[jira] [Created] (SPARK-13902) Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.

2016-03-15 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-13902: - Summary: Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first. Key: SPARK-13902 URL: https://issues.apache.org

[jira] [Created] (SPARK-13640) Synchronize ScalaReflection.mirror method.

2016-03-02 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-13640: - Summary: Synchronize ScalaReflection.mirror method. Key: SPARK-13640 URL: https://issues.apache.org/jira/browse/SPARK-13640 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-13357) Use generated projection and ordering for TakeOrderedAndProjectNode

2016-02-16 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-13357: - Summary: Use generated projection and ordering for TakeOrderedAndProjectNode Key: SPARK-13357 URL: https://issues.apache.org/jira/browse/SPARK-13357 Project: Spark

[jira] [Created] (SPARK-12976) Add LazilyGenerateOrdering and use it for RangePartitioner of Exchange.

2016-01-24 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-12976: - Summary: Add LazilyGenerateOrdering and use it for RangePartitioner of Exchange. Key: SPARK-12976 URL: https://issues.apache.org/jira/browse/SPARK-12976 Project: Sp

[jira] [Created] (SPARK-8476) Setters inc/decDiskBytesSpilled in TaskMetrics should also be private.

2015-06-18 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-8476: Summary: Setters inc/decDiskBytesSpilled in TaskMetrics should also be private. Key: SPARK-8476 URL: https://issues.apache.org/jira/browse/SPARK-8476 Project: Spark

[jira] [Created] (SPARK-7826) Suppress extra calling getCacheLocs.

2015-05-22 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-7826: Summary: Suppress extra calling getCacheLocs. Key: SPARK-7826 URL: https://issues.apache.org/jira/browse/SPARK-7826 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-5457) Add missing DSL for ApproxCountDistinct.

2015-01-28 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-5457: Summary: Add missing DSL for ApproxCountDistinct. Key: SPARK-5457 URL: https://issues.apache.org/jira/browse/SPARK-5457 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4720) Remainder should also return null if the divider is 0.

2014-12-03 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4720: Summary: Remainder should also return null if the divider is 0. Key: SPARK-4720 URL: https://issues.apache.org/jira/browse/SPARK-4720 Project: Spark Issue Ty

[jira] [Closed] (SPARK-4418) Remove restriction of binary arithmetic operations for fixed-precision decimal.

2014-11-18 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin closed SPARK-4418. Resolution: Not a Problem > Remove restriction of binary arithmetic operations for fixed-precision

[jira] [Created] (SPARK-4429) Build for Scala 2.11 using sbt fails.

2014-11-15 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4429: Summary: Build for Scala 2.11 using sbt fails. Key: SPARK-4429 URL: https://issues.apache.org/jira/browse/SPARK-4429 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4428) Use ${scala.binary.version} property for artifactId.

2014-11-15 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4428: Summary: Use ${scala.binary.version} property for artifactId. Key: SPARK-4428 URL: https://issues.apache.org/jira/browse/SPARK-4428 Project: Spark Issue Type

[jira] [Updated] (SPARK-4425) Handle NaN or Infinity cast to Timestamp correctly

2014-11-15 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-4425: - Summary: Handle NaN or Infinity cast to Timestamp correctly (was: Handle NaN cast to Timestamp co

[jira] [Created] (SPARK-4425) Handle NaN cast to Timestamp correctly

2014-11-15 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4425: Summary: Handle NaN cast to Timestamp correctly Key: SPARK-4425 URL: https://issues.apache.org/jira/browse/SPARK-4425 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4420) Change nullability of Cast from DoubleType/FloatType to DecimalType.

2014-11-14 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4420: Summary: Change nullability of Cast from DoubleType/FloatType to DecimalType. Key: SPARK-4420 URL: https://issues.apache.org/jira/browse/SPARK-4420 Project: Spark

[jira] [Created] (SPARK-4418) Remove restriction of binary arithmetic operations for fixed-precision decimal.

2014-11-14 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4418: Summary: Remove restriction of binary arithmetic operations for fixed-precision decimal. Key: SPARK-4418 URL: https://issues.apache.org/jira/browse/SPARK-4418 Project

[jira] [Created] (SPARK-4319) Enable an ignored test "null count".

2014-11-10 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4319: Summary: Enable an ignored test "null count". Key: SPARK-4319 URL: https://issues.apache.org/jira/browse/SPARK-4319 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-4318) Fix empty sum distinct.

2014-11-10 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4318: Summary: Fix empty sum distinct. Key: SPARK-4318 URL: https://issues.apache.org/jira/browse/SPARK-4318 Project: Spark Issue Type: Bug Components: S

[jira] [Created] (SPARK-4293) Make Cast be able to handle complex types.

2014-11-06 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4293: Summary: Make Cast be able to handle complex types. Key: SPARK-4293 URL: https://issues.apache.org/jira/browse/SPARK-4293 Project: Spark Issue Type: Improvem

[jira] [Created] (SPARK-4270) Fix Cast from DateType to DecimalType.

2014-11-06 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4270: Summary: Fix Cast from DateType to DecimalType. Key: SPARK-4270 URL: https://issues.apache.org/jira/browse/SPARK-4270 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-4245) Fix containsNull of the result ArrayType of CreateArray expression.

2014-11-05 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4245: Summary: Fix containsNull of the result ArrayType of CreateArray expression. Key: SPARK-4245 URL: https://issues.apache.org/jira/browse/SPARK-4245 Project: Spark

[jira] [Created] (SPARK-4193) Disable doclint in Java 8 to prevent from build error.

2014-11-01 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-4193: Summary: Disable doclint in Java 8 to prevent from build error. Key: SPARK-4193 URL: https://issues.apache.org/jira/browse/SPARK-4193 Project: Spark Issue Ty

[jira] [Created] (SPARK-3986) Fix package names to fit their directory names.

2014-10-16 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-3986: Summary: Fix package names to fit their directory names. Key: SPARK-3986 URL: https://issues.apache.org/jira/browse/SPARK-3986 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-3969) Optimizer should have a super class as an interface.

2014-10-16 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-3969: Summary: Optimizer should have a super class as an interface. Key: SPARK-3969 URL: https://issues.apache.org/jira/browse/SPARK-3969 Project: Spark Issue Type

[jira] [Created] (SPARK-3966) Fix nullabilities of Cast related to DateType.

2014-10-15 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-3966: Summary: Fix nullabilities of Cast related to DateType. Key: SPARK-3966 URL: https://issues.apache.org/jira/browse/SPARK-3966 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-3771) AppendingParquetOutputFormat should use reflection to prevent from breaking binary-compatibility.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-3771: - Summary: AppendingParquetOutputFormat should use reflection to prevent from breaking binary-compat

[jira] [Closed] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin closed SPARK-3764. Resolution: Fixed > Invalid dependencies of artifacts in Maven Central Repository. > ---

[jira] [Commented] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157447#comment-14157447 ] Takuya Ueshin commented on SPARK-3764: -- I filed a new issue SPARK-3771 and close this

[jira] [Created] (SPARK-3771) AppendingParquetOutputFormat should use reflection to prevent breaking binary-compatibility.

2014-10-02 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-3771: Summary: AppendingParquetOutputFormat should use reflection to prevent breaking binary-compatibility. Key: SPARK-3771 URL: https://issues.apache.org/jira/browse/SPARK-3771

[jira] [Commented] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157418#comment-14157418 ] Takuya Ueshin commented on SPARK-3764: -- {{AppendingParquetOutputFormat}} is using {{T

[jira] [Commented] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156599#comment-14156599 ] Takuya Ueshin commented on SPARK-3764: -- Ah, I see that {{context.getTaskAttemptID}} a

[jira] [Commented] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156588#comment-14156588 ] Takuya Ueshin commented on SPARK-3764: -- But there are some codes using binary-incompa

[jira] [Commented] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156578#comment-14156578 ] Takuya Ueshin commented on SPARK-3764: -- Now I found the instruction [here|http://spa

[jira] [Commented] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156576#comment-14156576 ] Takuya Ueshin commented on SPARK-3764: -- Ah, so, these artifacts are only for hadoop-2

[jira] [Created] (SPARK-3764) Invalid dependencies of artifacts in Maven Central Repository.

2014-10-02 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-3764: Summary: Invalid dependencies of artifacts in Maven Central Repository. Key: SPARK-3764 URL: https://issues.apache.org/jira/browse/SPARK-3764 Project: Spark

[jira] [Commented] (SPARK-3341) The dataType of Sqrt expression should be DoubleType.

2014-09-02 Thread Takuya Ueshin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119273#comment-14119273 ] Takuya Ueshin commented on SPARK-3341: -- Hi, this issue's Fix Version should be the sa

[jira] [Created] (SPARK-3341) The dataType of Sqrt expression should be DoubleType.

2014-09-01 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-3341: Summary: The dataType of Sqrt expression should be DoubleType. Key: SPARK-3341 URL: https://issues.apache.org/jira/browse/SPARK-3341 Project: Spark Issue Typ

<    3   4   5   6   7   8   9   >