[jira] [Commented] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1594#comment-1594 ] Kazuaki Ishizaki commented on SPARK-20112: -- [~MasterDDT] Thank you for preparing additional

[jira] [Commented] (SPARK-20112) SIGSEGV in GeneratedIterator.sort_addToSorter

[ https://issues.apache.org/jira/browse/SPARK-20112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945389#comment-15945389 ] Kazuaki Ishizaki commented on SPARK-20112: -- SPARK-18745 fixed integer overflow issues in

[jira] [Created] (SPARK-20101) Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is set to "true"

Kazuaki Ishizaki created SPARK-20101: Summary: Use OffHeapColumnVector when "spark.memory.offHeap.enabled" is set to "true" Key: SPARK-20101 URL: https://issues.apache.org/jira/browse/SPARK-20101

[jira] [Commented] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15939511#comment-15939511 ] Kazuaki Ishizaki commented on SPARK-19372: -- I implemented the code to take care of it, and am

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937907#comment-15937907 ] Kazuaki Ishizaki commented on SPARK-14083: -- I agree with you. I do not think that current status

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937886#comment-15937886 ] Kazuaki Ishizaki commented on SPARK-14083: -- [~viirya] For a while, I will be able to update the

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937883#comment-15937883 ] Kazuaki Ishizaki commented on SPARK-14083: -- [~maropu] Thanks. > Analyze JVM bytecode and turn

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937783#comment-15937783 ] Kazuaki Ishizaki commented on SPARK-14083: -- [~viirya] Thank you for your comment. Good to hear.

[jira] [Updated] (SPARK-20046) Facilitate loop optimizations in a JIT compiler regarding sqlContext.read.parquet()

[ https://issues.apache.org/jira/browse/SPARK-20046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-20046: - Issue Type: Improvement (was: Bug) > Facilitate loop optimizations in a JIT compiler

[jira] [Created] (SPARK-20046) Facilitate loop optimizations in a JIT compiler regarding sqlContext.read.parquet()

Kazuaki Ishizaki created SPARK-20046: Summary: Facilitate loop optimizations in a JIT compiler regarding sqlContext.read.parquet() Key: SPARK-20046 URL: https://issues.apache.org/jira/browse/SPARK-20046

[jira] [Comment Edited] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929399#comment-15929399 ] Kazuaki Ishizaki edited comment on SPARK-19984 at 3/17/17 4:23 AM: ---

[jira] [Commented] (SPARK-19984) ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java'

[ https://issues.apache.org/jira/browse/SPARK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929399#comment-15929399 ] Kazuaki Ishizaki commented on SPARK-19984: -- This problem occurs since Spark generates

[jira] [Created] (SPARK-19959) df[java.lang.Long].collect throws NullPointerException if df includes null

Kazuaki Ishizaki created SPARK-19959: Summary: df[java.lang.Long].collect throws NullPointerException if df includes null Key: SPARK-19959 URL: https://issues.apache.org/jira/browse/SPARK-19959

[jira] [Commented] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

[ https://issues.apache.org/jira/browse/SPARK-19950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925357#comment-15925357 ] Kazuaki Ishizaki commented on SPARK-19950: -- [~hyukjin.kwon] Thank you for pointing out

[jira] [Created] (SPARK-19950) nullable ignored when df.load() is executed for file-based data source

Kazuaki Ishizaki created SPARK-19950: Summary: nullable ignored when df.load() is executed for file-based data source Key: SPARK-19950 URL: https://issues.apache.org/jira/browse/SPARK-19950

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904460#comment-15904460 ] Kazuaki Ishizaki commented on SPARK-14083: -- I rebased this with master:

[jira] [Commented] (SPARK-19875) Map->filter on many columns gets stuck in constraint inference optimization code

[ https://issues.apache.org/jira/browse/SPARK-19875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903584#comment-15903584 ] Kazuaki Ishizaki commented on SPARK-19875: -- I got the following stack trace. This stuck seems to

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896452#comment-15896452 ] Kazuaki Ishizaki commented on SPARK-14083: -- Does anyone go forward with this? If not, I will

[jira] [Commented] (SPARK-19503) Execution Plan Optimizer: avoid sort or shuffle when it does not change end result such as df.sort(...).count()

[ https://issues.apache.org/jira/browse/SPARK-19503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895740#comment-15895740 ] Kazuaki Ishizaki commented on SPARK-19503: -- Is it better to control whether we prune local and

[jira] [Commented] (SPARK-19503) Execution Plan Optimizer: avoid sort or shuffle when it does not change end result such as df.sort(...).count()

[ https://issues.apache.org/jira/browse/SPARK-19503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891699#comment-15891699 ] Kazuaki Ishizaki commented on SPARK-19503: -- If it is good to leave sort intact for now, do we

[jira] [Commented] (SPARK-19468) Dataset slow because of unnecessary shuffles

[ https://issues.apache.org/jira/browse/SPARK-19468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891658#comment-15891658 ] Kazuaki Ishizaki commented on SPARK-19468: -- Interesting. For {{val joined1 = ds1.joinWith(ds2,

[jira] [Comment Edited] (SPARK-19741) ClassCastException when using Dataset with type containing value types

[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891548#comment-15891548 ] Kazuaki Ishizaki edited comment on SPARK-19741 at 3/2/17 3:17 AM: -- I am

[jira] [Commented] (SPARK-19741) ClassCastException when using Dataset with type containing value types

[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891548#comment-15891548 ] Kazuaki Ishizaki commented on SPARK-19741: -- I am afraid whether my sample program succeeded to

[jira] [Commented] (SPARK-19741) ClassCastException when using Dataset with type containing value types

[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890712#comment-15890712 ] Kazuaki Ishizaki commented on SPARK-19741: -- The following program causes an exception regarding

[jira] [Updated] (SPARK-19786) Facilitate loop optimizations in a JIT compiler regarding range()

[ https://issues.apache.org/jira/browse/SPARK-19786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-19786: - Summary: Facilitate loop optimizations in a JIT compiler regarding range() (was:

[jira] [Created] (SPARK-19786) Facilitate loop optimization in a JIT compiler regarding range()

Kazuaki Ishizaki created SPARK-19786: Summary: Facilitate loop optimization in a JIT compiler regarding range() Key: SPARK-19786 URL: https://issues.apache.org/jira/browse/SPARK-19786 Project:

[jira] [Commented] (SPARK-19741) ClassCastException when using Dataset with type containing value types

[ https://issues.apache.org/jira/browse/SPARK-19741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885171#comment-15885171 ] Kazuaki Ishizaki commented on SPARK-19741: -- Would it be possible to attache a pair of the error

[jira] [Commented] (SPARK-15678) Not use cache on appends and overwrites

[ https://issues.apache.org/jira/browse/SPARK-15678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883035#comment-15883035 ] Kazuaki Ishizaki commented on SPARK-15678: -- Sorry for being late to reply. According to the

[jira] [Commented] (SPARK-15678) Not use cache on appends and overwrites

[ https://issues.apache.org/jira/browse/SPARK-15678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877575#comment-15877575 ] Kazuaki Ishizaki commented on SPARK-15678: -- How about insert {{spark.catalog.refreshByPath()}}

[jira] [Comment Edited] (SPARK-15678) Not use cache on appends and overwrites

[ https://issues.apache.org/jira/browse/SPARK-15678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877575#comment-15877575 ] Kazuaki Ishizaki edited comment on SPARK-15678 at 2/22/17 6:37 AM: --- How

[jira] [Commented] (SPARK-19653) `Vector` Type Should Be A First-Class Citizen In Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-19653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872901#comment-15872901 ] Kazuaki Ishizaki commented on SPARK-19653: -- cc: [~cloud_fan] > `Vector` Type Should Be A

[jira] [Resolved] (SPARK-16043) Prepare GenericArrayData implementation specialized for a primitive array

[ https://issues.apache.org/jira/browse/SPARK-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-16043. -- Resolution: Fixed Fix Version/s: 2.2.0 > Prepare GenericArrayData

[jira] [Commented] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851472#comment-15851472 ] Kazuaki Ishizaki commented on SPARK-19372: -- I was able to reproduce this. I am thinking how to

[jira] [Commented] (SPARK-19008) Avoid boxing/unboxing overhead of calling a lambda with primitive type from Dataset program

[ https://issues.apache.org/jira/browse/SPARK-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779544#comment-15779544 ] Kazuaki Ishizaki commented on SPARK-19008: -- I will work for this > Avoid boxing/unboxing

[jira] [Updated] (SPARK-19008) Avoid boxing/unboxing overhead of calling a lambda with primitive type from Dataset program

[ https://issues.apache.org/jira/browse/SPARK-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-19008: - Description: In a

[jira] [Created] (SPARK-19008) Avoid boxing/unboxing overhead of calling a lambda with primitive type from Dataset program

Kazuaki Ishizaki created SPARK-19008: Summary: Avoid boxing/unboxing overhead of calling a lambda with primitive type from Dataset program Key: SPARK-19008 URL:

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779288#comment-15779288 ] Kazuaki Ishizaki commented on SPARK-14083: --

[jira] [Commented] (SPARK-18859) Catalyst codegen does not mark column as nullable when it should. Causes NPE

[ https://issues.apache.org/jira/browse/SPARK-18859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764789#comment-15764789 ] Kazuaki Ishizaki commented on SPARK-18859: -- I think that this is an issue in join operation.

[jira] [Commented] (SPARK-18814) CheckAnalysis rejects TPCDS query 32

[ https://issues.apache.org/jira/browse/SPARK-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742828#comment-15742828 ] Kazuaki Ishizaki commented on SPARK-18814: -- I found the same error

[jira] [Commented] (SPARK-16073) Performance of Parquet encodings on saving primitive arrays

[ https://issues.apache.org/jira/browse/SPARK-16073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740807#comment-15740807 ] Kazuaki Ishizaki commented on SPARK-16073: -- It is an interesting topic. In the current

[jira] [Comment Edited] (SPARK-18745) java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735580#comment-15735580 ] Kazuaki Ishizaki edited comment on SPARK-18745 at 12/9/16 3:29 PM: --- I

[jira] [Comment Edited] (SPARK-18745) java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735580#comment-15735580 ] Kazuaki Ishizaki edited comment on SPARK-18745 at 12/9/16 3:21 PM: --- I

[jira] [Commented] (SPARK-18745) java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735580#comment-15735580 ] Kazuaki Ishizaki commented on SPARK-18745: -- I identified a root cause of this

[jira] [Updated] (SPARK-18745) java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-18745: - Affects Version/s: 2.2.0 > java.lang.IndexOutOfBoundsException running query 68 Spark

[jira] [Commented] (SPARK-18745) java.lang.IndexOutOfBoundsException running query 68 Spark SQL on (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726486#comment-15726486 ] Kazuaki Ishizaki commented on SPARK-18745: -- I work with [~jfc...@us.ibm.com] >

[jira] [Created] (SPARK-18653) Dataset.show() generates incorrect padding for Unicode Character

Kazuaki Ishizaki created SPARK-18653: Summary: Dataset.show() generates incorrect padding for Unicode Character Key: SPARK-18653 URL: https://issues.apache.org/jira/browse/SPARK-18653 Project:

[jira] [Commented] (SPARK-17680) Unicode Character Support for Column Names and Comments

[ https://issues.apache.org/jira/browse/SPARK-17680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707629#comment-15707629 ] Kazuaki Ishizaki commented on SPARK-17680: -- Sorry, it is my mistake. > Unicode Character

[jira] [Commented] (SPARK-18502) Spark does not handle columns that contain backquote (`)

[ https://issues.apache.org/jira/browse/SPARK-18502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706385#comment-15706385 ] Kazuaki Ishizaki commented on SPARK-18502: -- I can reproduce this exception using the following

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702760#comment-15702760 ] Kazuaki Ishizaki commented on SPARK-18492: -- I realized that the following code can reproduce

[jira] [Resolved] (SPARK-15950) Eliminate unreachable code at projection for complex types

[ https://issues.apache.org/jira/browse/SPARK-15950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-15950. -- Resolution: Duplicate > Eliminate unreachable code at projection for complex types >

[jira] [Comment Edited] (SPARK-18492) GeneratedIterator grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674599#comment-15674599 ] Kazuaki Ishizaki edited comment on SPARK-18492 at 11/17/16 7:31 PM: I

[jira] [Commented] (SPARK-18492) GeneratedIterator grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674599#comment-15674599 ] Kazuaki Ishizaki commented on SPARK-18492: -- Can you post a small program that can reproduce this

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671237#comment-15671237 ] Kazuaki Ishizaki commented on SPARK-18458: -- I worked with [~jfc...@us.ibm.com]. Then, I

[jira] [Issue Comment Deleted] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-18458: - Comment: was deleted (was: I worked with [~jfc...@us.ibm.com]. Then, I identified that a

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15671229#comment-15671229 ] Kazuaki Ishizaki commented on SPARK-18458: -- I worked with [~jfc...@us.ibm.com]. Then, I

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15670828#comment-15670828 ] Kazuaki Ishizaki commented on SPARK-18458: -- I see. I will do that. > core dumped running Spark

[jira] [Commented] (SPARK-18458) core dumped running Spark SQL on large data volume (100TB)

[ https://issues.apache.org/jira/browse/SPARK-18458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668948#comment-15668948 ] Kazuaki Ishizaki commented on SPARK-18458: -- I work for this. > core dumped running Spark SQL on

[jira] [Updated] (SPARK-18284) Scheme of DataFrame generated from RDD is diffrent between master and 2.0

[ https://issues.apache.org/jira/browse/SPARK-18284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-18284: - Description: When the following program is executed, a schema of dataframe is different

[jira] [Updated] (SPARK-18284) Scheme of DataFrame generated from RDD is diffrent between master and 2.0

[ https://issues.apache.org/jira/browse/SPARK-18284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-18284: - Affects Version/s: 2.1.0 > Scheme of DataFrame generated from RDD is diffrent between

[jira] [Created] (SPARK-18284) Scheme of DataFrame generated from RDD is diffrent between master and 2.0

Kazuaki Ishizaki created SPARK-18284: Summary: Scheme of DataFrame generated from RDD is diffrent between master and 2.0 Key: SPARK-18284 URL: https://issues.apache.org/jira/browse/SPARK-18284

[jira] [Commented] (SPARK-18207) class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-18207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629522#comment-15629522 ] Kazuaki Ishizaki commented on SPARK-18207: -- I created a smaller program to reproduce this

[jira] [Commented] (SPARK-18125) Spark generated code causes CompileException when groupByKey, reduceGroups and map(_._2) are used

[ https://issues.apache.org/jira/browse/SPARK-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615034#comment-15615034 ] Kazuaki Ishizaki commented on SPARK-18125: -- I confirmed this code can reproduce on 2.0.1. This

[jira] [Commented] (SPARK-18147) Broken Spark SQL Codegen

[ https://issues.apache.org/jira/browse/SPARK-18147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614959#comment-15614959 ] Kazuaki Ishizaki commented on SPARK-18147: -- This also cause the same exception. {code:java}

[jira] [Commented] (SPARK-15687) Columnar execution engine

[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15588641#comment-15588641 ] Kazuaki Ishizaki commented on SPARK-15687: -- [#15219|https://github.com/apache/spark/pull/15219]

[jira] [Created] (SPARK-17915) Prepare ColumnVector implementation for UnsafeData

Kazuaki Ishizaki created SPARK-17915: Summary: Prepare ColumnVector implementation for UnsafeData Key: SPARK-17915 URL: https://issues.apache.org/jira/browse/SPARK-17915 Project: Spark

[jira] [Created] (SPARK-17912) Refactor code generation to get data for ColumnVector/ColumnarBatch

Kazuaki Ishizaki created SPARK-17912: Summary: Refactor code generation to get data for ColumnVector/ColumnarBatch Key: SPARK-17912 URL: https://issues.apache.org/jira/browse/SPARK-17912 Project:

[jira] [Created] (SPARK-17905) Added test cases for InMemoryRelation

Kazuaki Ishizaki created SPARK-17905: Summary: Added test cases for InMemoryRelation Key: SPARK-17905 URL: https://issues.apache.org/jira/browse/SPARK-17905 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569493#comment-15569493 ] Kazuaki Ishizaki commented on SPARK-16845: -- Thank you for preparing the case. I noticed that the

[jira] [Issue Comment Deleted] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16845: - Comment: was deleted (was: Thank you for preparing the case. I noticed that the

[jira] [Commented] (SPARK-16845) org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering" grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-16845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569489#comment-15569489 ] Kazuaki Ishizaki commented on SPARK-16845: -- Thank you for preparing the case. I noticed that the

[jira] [Resolved] (SPARK-16223) Codegen failure with a Dataframe program using an array

[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-16223. -- Resolution: Fixed > Codegen failure with a Dataframe program using an array >

[jira] [Commented] (SPARK-16223) Codegen failure with a Dataframe program using an array

[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557853#comment-15557853 ] Kazuaki Ishizaki commented on SPARK-16223: -- When I rerun it with {{commit

[jira] [Created] (SPARK-17490) Optimize SerializeFromObject for primitive array

Kazuaki Ishizaki created SPARK-17490: Summary: Optimize SerializeFromObject for primitive array Key: SPARK-17490 URL: https://issues.apache.org/jira/browse/SPARK-17490 Project: Spark

[jira] [Updated] (SPARK-16213) Reduce runtime overhead of a program that creates an primitive array in DataFrame

[ https://issues.apache.org/jira/browse/SPARK-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16213: - Affects Version/s: 2.0.0 > Reduce runtime overhead of a program that creates an

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422757#comment-15422757 ] Kazuaki Ishizaki commented on SPARK-15285: -- This is because the original pull request breaks a

[jira] [Reopened] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reopened SPARK-15285: -- This problem can be reproduced at commit id 7de30d6e9e5d3020d2ba8c2ce08893d9cd822b56. >

[jira] [Commented] (SPARK-16913) [SQL] Better codegen where querying nested struct

[ https://issues.apache.org/jira/browse/SPARK-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410840#comment-15410840 ] Kazuaki Ishizaki commented on SPARK-16913: -- It seems to copy each elements in a struct. Since

[jira] [Created] (SPARK-16902) Custom ExpressionEncoder for primitive array is not effective

Kazuaki Ishizaki created SPARK-16902: Summary: Custom ExpressionEncoder for primitive array is not effective Key: SPARK-16902 URL: https://issues.apache.org/jira/browse/SPARK-16902 Project: Spark

[jira] [Commented] (SPARK-16807) Optimize some ABS() statements

[ https://issues.apache.org/jira/browse/SPARK-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400532#comment-15400532 ] Kazuaki Ishizaki commented on SPARK-16807: -- Interesting if we can ensure {{x - y}} is not

[jira] [Updated] (SPARK-16223) Codegen failure with a Dataframe program using an array

[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16223: - Description: When we compile a Dataframe program with an operation to large array,

[jira] [Comment Edited] (SPARK-15687) Columnar execution engine

[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376999#comment-15376999 ] Kazuaki Ishizaki edited comment on SPARK-15687 at 7/14/16 2:25 PM: --- It

[jira] [Commented] (SPARK-15687) Columnar execution engine

[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376999#comment-15376999 ] Kazuaki Ishizaki commented on SPARK-15687: -- It would be good to introduce trait for

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369703#comment-15369703 ] Kazuaki Ishizaki commented on SPARK-15467: -- [Janino

[jira] [Updated] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

[ https://issues.apache.org/jira/browse/SPARK-16412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16412: - Issue Type: Improvement (was: Bug) > Generate Java code that gets an array in each

[jira] [Created] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

Kazuaki Ishizaki created SPARK-16412: Summary: Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called Key: SPARK-16412 URL:

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360752#comment-15360752 ] Kazuaki Ishizaki commented on SPARK-15467: -- My

[jira] [Created] (SPARK-16223) Codegen failure with a Dataframe program using an array

Kazuaki Ishizaki created SPARK-16223: Summary: Codegen failure with a Dataframe program using an array Key: SPARK-16223 URL: https://issues.apache.org/jira/browse/SPARK-16223 Project: Spark

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350005#comment-15350005 ] Kazuaki Ishizaki commented on SPARK-16070: -- I added two JIRA entries, which address DataFrame

[jira] [Created] (SPARK-16215) Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset

Kazuaki Ishizaki created SPARK-16215: Summary: Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset Key: SPARK-16215 URL: https://issues.apache.org/jira/browse/SPARK-16215

[jira] [Created] (SPARK-16213) Reduce runtime overhead of a program that creates an primitive array in DataFrame

Kazuaki Ishizaki created SPARK-16213: Summary: Reduce runtime overhead of a program that creates an primitive array in DataFrame Key: SPARK-16213 URL: https://issues.apache.org/jira/browse/SPARK-16213

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345620#comment-15345620 ] Kazuaki Ishizaki commented on SPARK-15899: -- I think so. As [~sowen] proposed, we may need a

[jira] [Comment Edited] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341427#comment-15341427 ] Kazuaki Ishizaki edited comment on SPARK-16070 at 6/21/16 9:22 AM: ---

[jira] [Comment Edited] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341408#comment-15341408 ] Kazuaki Ishizaki edited comment on SPARK-16070 at 6/21/16 9:22 AM: ---

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341427#comment-15341427 ] Kazuaki Ishizaki commented on SPARK-16070: -- Other JIRAs for DataFrame issues with primitive

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341408#comment-15341408 ] Kazuaki Ishizaki commented on SPARK-16070: -- [~mengxr], thank you for creating an umbrella. I

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341376#comment-15341376 ] Kazuaki Ishizaki commented on SPARK-15467: -- Thank you for letting me know it. Now, it looks [an

[jira] [Created] (SPARK-16043) Prepare GenericArrayData implementation specialized for a primitive array

Kazuaki Ishizaki created SPARK-16043: Summary: Prepare GenericArrayData implementation specialized for a primitive array Key: SPARK-16043 URL: https://issues.apache.org/jira/browse/SPARK-16043

[jira] [Created] (SPARK-16042) Eliminate nullcheck code at projection for an array type

Kazuaki Ishizaki created SPARK-16042: Summary: Eliminate nullcheck code at projection for an array type Key: SPARK-16042 URL: https://issues.apache.org/jira/browse/SPARK-16042 Project: Spark

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334443#comment-15334443 ] Kazuaki Ishizaki commented on SPARK-15467: -- We are waiting for author's review at

[jira] [Created] (SPARK-15985) Reduce runtime overhead of a program that reads an primitive array in Dataset

Kazuaki Ishizaki created SPARK-15985: Summary: Reduce runtime overhead of a program that reads an primitive array in Dataset Key: SPARK-15985 URL: https://issues.apache.org/jira/browse/SPARK-15985

<    4   5   6   7   8   9   10   >