[jira] [Resolved] (SPARK-16223) Codegen failure with a Dataframe program using an array

2016-10-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-16223. -- Resolution: Fixed > Codegen failure with a Dataframe program using an array >

[jira] [Commented] (SPARK-16223) Codegen failure with a Dataframe program using an array

2016-10-08 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557853#comment-15557853 ] Kazuaki Ishizaki commented on SPARK-16223: -- When I rerun it with {{commit

[jira] [Created] (SPARK-17490) Optimize SerializeFromObject for primitive array

2016-09-10 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-17490: Summary: Optimize SerializeFromObject for primitive array Key: SPARK-17490 URL: https://issues.apache.org/jira/browse/SPARK-17490 Project: Spark

[jira] [Updated] (SPARK-16213) Reduce runtime overhead of a program that creates an primitive array in DataFrame

2016-09-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16213: - Affects Version/s: 2.0.0 > Reduce runtime overhead of a program that creates an

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-08-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422757#comment-15422757 ] Kazuaki Ishizaki commented on SPARK-15285: -- This is because the original pull request breaks a

[jira] [Reopened] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-08-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki reopened SPARK-15285: -- This problem can be reproduced at commit id 7de30d6e9e5d3020d2ba8c2ce08893d9cd822b56. >

[jira] [Commented] (SPARK-16913) [SQL] Better codegen where querying nested struct

2016-08-07 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410840#comment-15410840 ] Kazuaki Ishizaki commented on SPARK-16913: -- It seems to copy each elements in a struct. Since

[jira] [Created] (SPARK-16902) Custom ExpressionEncoder for primitive array is not effective

2016-08-04 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16902: Summary: Custom ExpressionEncoder for primitive array is not effective Key: SPARK-16902 URL: https://issues.apache.org/jira/browse/SPARK-16902 Project: Spark

[jira] [Commented] (SPARK-16807) Optimize some ABS() statements

2016-07-30 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400532#comment-15400532 ] Kazuaki Ishizaki commented on SPARK-16807: -- Interesting if we can ensure {{x - y}} is not

[jira] [Updated] (SPARK-16223) Codegen failure with a Dataframe program using an array

2016-07-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16223: - Description: When we compile a Dataframe program with an operation to large array,

[jira] [Comment Edited] (SPARK-15687) Columnar execution engine

2016-07-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376999#comment-15376999 ] Kazuaki Ishizaki edited comment on SPARK-15687 at 7/14/16 2:25 PM: --- It

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-07-14 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376999#comment-15376999 ] Kazuaki Ishizaki commented on SPARK-15687: -- It would be good to introduce trait for

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-07-10 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15369703#comment-15369703 ] Kazuaki Ishizaki commented on SPARK-15467: -- [Janino

[jira] [Updated] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

2016-07-07 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-16412: - Issue Type: Improvement (was: Bug) > Generate Java code that gets an array in each

[jira] [Created] (SPARK-16412) Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called

2016-07-07 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16412: Summary: Generate Java code that gets an array in each column of CachedBatch when DataFrame.cache() is called Key: SPARK-16412 URL:

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-07-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360752#comment-15360752 ] Kazuaki Ishizaki commented on SPARK-15467: -- My

[jira] [Created] (SPARK-16223) Codegen failure with a Dataframe program using an array

2016-06-27 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16223: Summary: Codegen failure with a Dataframe program using an array Key: SPARK-16223 URL: https://issues.apache.org/jira/browse/SPARK-16223 Project: Spark

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-26 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350005#comment-15350005 ] Kazuaki Ishizaki commented on SPARK-16070: -- I added two JIRA entries, which address DataFrame

[jira] [Created] (SPARK-16215) Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset

2016-06-25 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16215: Summary: Reduce runtime overhead of a program that writes an primitive array in Dataframe/Dataset Key: SPARK-16215 URL: https://issues.apache.org/jira/browse/SPARK-16215

[jira] [Created] (SPARK-16213) Reduce runtime overhead of a program that creates an primitive array in DataFrame

2016-06-25 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16213: Summary: Reduce runtime overhead of a program that creates an primitive array in DataFrame Key: SPARK-16213 URL: https://issues.apache.org/jira/browse/SPARK-16213

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345620#comment-15345620 ] Kazuaki Ishizaki commented on SPARK-15899: -- I think so. As [~sowen] proposed, we may need a

[jira] [Comment Edited] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341427#comment-15341427 ] Kazuaki Ishizaki edited comment on SPARK-16070 at 6/21/16 9:22 AM: ---

[jira] [Comment Edited] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341408#comment-15341408 ] Kazuaki Ishizaki edited comment on SPARK-16070 at 6/21/16 9:22 AM: ---

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341427#comment-15341427 ] Kazuaki Ishizaki commented on SPARK-16070: -- Other JIRAs for DataFrame issues with primitive

[jira] [Commented] (SPARK-16070) DataFrame/Parquet issues with primitive arrays

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341408#comment-15341408 ] Kazuaki Ishizaki commented on SPARK-16070: -- [~mengxr], thank you for creating an umbrella. I

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-06-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341376#comment-15341376 ] Kazuaki Ishizaki commented on SPARK-15467: -- Thank you for letting me know it. Now, it looks [an

[jira] [Created] (SPARK-16043) Prepare GenericArrayData implementation specialized for a primitive array

2016-06-18 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16043: Summary: Prepare GenericArrayData implementation specialized for a primitive array Key: SPARK-16043 URL: https://issues.apache.org/jira/browse/SPARK-16043

[jira] [Created] (SPARK-16042) Eliminate nullcheck code at projection for an array type

2016-06-18 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-16042: Summary: Eliminate nullcheck code at projection for an array type Key: SPARK-16042 URL: https://issues.apache.org/jira/browse/SPARK-16042 Project: Spark

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-06-16 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334443#comment-15334443 ] Kazuaki Ishizaki commented on SPARK-15467: -- We are waiting for author's review at

[jira] [Created] (SPARK-15985) Reduce runtime overhead of a program that reads an primitive array in Dataset

2016-06-16 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15985: Summary: Reduce runtime overhead of a program that reads an primitive array in Dataset Key: SPARK-15985 URL: https://issues.apache.org/jira/browse/SPARK-15985

[jira] [Created] (SPARK-15962) Introduce additonal implementation with a dense format for UnsafeArrayData

2016-06-15 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15962: Summary: Introduce additonal implementation with a dense format for UnsafeArrayData Key: SPARK-15962 URL: https://issues.apache.org/jira/browse/SPARK-15962

[jira] [Created] (SPARK-15950) Eliminate unreachable code at projection for complex types

2016-06-14 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15950: Summary: Eliminate unreachable code at projection for complex types Key: SPARK-15950 URL: https://issues.apache.org/jira/browse/SPARK-15950 Project: Spark

[jira] [Issue Comment Deleted] (SPARK-15899) file scheme should be used correctly

2016-06-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-15899: - Comment: was deleted (was: When I added the two extra slashes, it works on Linux. But,

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327825#comment-15327825 ] Kazuaki Ishizaki commented on SPARK-15899: -- When I added the two extra slashes, it works on

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-13 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15327824#comment-15327824 ] Kazuaki Ishizaki commented on SPARK-15899: -- When I added the two extra slashes, it works on

[jira] [Commented] (SPARK-15899) file scheme should be used correctly

2016-06-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15326537#comment-15326537 ] Kazuaki Ishizaki commented on SPARK-15899: -- Thank you for your comments. It is a little bit

[jira] [Created] (SPARK-15899) file scheme should be used correctly

2016-06-11 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15899: Summary: file scheme should be used correctly Key: SPARK-15899 URL: https://issues.apache.org/jira/browse/SPARK-15899 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-15687) Columnar execution engine

2016-06-06 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316858#comment-15316858 ] Kazuaki Ishizaki edited comment on SPARK-15687 at 6/6/16 5:49 PM: -- Thank

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-06-06 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316858#comment-15316858 ] Kazuaki Ishizaki commented on SPARK-15687: -- Thank you for your answers * How we pass columnar

[jira] [Commented] (SPARK-15687) Columnar execution engine

2016-06-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15310663#comment-15310663 ] Kazuaki Ishizaki commented on SPARK-15687: -- Thank you for creating interesting JIRA entry. Based

[jira] [Commented] (SPARK-15258) Nested/Chained case statements generate codegen over 64k exception

2016-05-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297896#comment-15297896 ] Kazuaki Ishizaki commented on SPARK-15258: -- {{commit d642b273544bb77ef7f584326aa2d214649ac61b}}

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-24 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15297784#comment-15297784 ] Kazuaki Ishizaki commented on SPARK-15467: -- I submitted [a pull

[jira] [Comment Edited] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296743#comment-15296743 ] Kazuaki Ishizaki edited comment on SPARK-15467 at 5/24/16 3:54 AM: --- The

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296929#comment-15296929 ] Kazuaki Ishizaki commented on SPARK-15467: -- I created test cases to reproduce this problem and a

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296743#comment-15296743 ] Kazuaki Ishizaki commented on SPARK-15467: -- The following code can reproduce the same issue.

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296102#comment-15296102 ] Kazuaki Ishizaki commented on SPARK-15467: -- I will try to submit a patch against Janino. >

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

2016-05-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295499#comment-15295499 ] Kazuaki Ishizaki commented on SPARK-15467: -- IMHO, this issue is due to Janino's implementation.

[jira] [Commented] (SPARK-15258) Nested/Chained case statements generate codegen over 64k exception

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295402#comment-15295402 ] Kazuaki Ishizaki commented on SPARK-15258: -- This PR is not for SPARK-15258. This is for

[jira] [Comment Edited] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295299#comment-15295299 ] Kazuaki Ishizaki edited comment on SPARK-15285 at 5/21/16 11:33 PM: I

[jira] [Comment Edited] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295299#comment-15295299 ] Kazuaki Ishizaki edited comment on SPARK-15285 at 5/21/16 11:12 PM: I

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295299#comment-15295299 ] Kazuaki Ishizaki commented on SPARK-15285: -- I created a

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-21 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295014#comment-15295014 ] Kazuaki Ishizaki commented on SPARK-15285: -- I see, I started doing this. > Generated

[jira] [Commented] (SPARK-15285) Generated SpecificSafeProjection.apply method grows beyond 64 KB

2016-05-20 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294494#comment-15294494 ] Kazuaki Ishizaki commented on SPARK-15285: -- I can take it today if they are busy. > Generated

[jira] [Created] (SPARK-15380) Generate code that stores a float/double value in each column from ColumnarBatch when DataFrame.cache() is used

2016-05-18 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15380: Summary: Generate code that stores a float/double value in each column from ColumnarBatch when DataFrame.cache() is used Key: SPARK-15380 URL:

[jira] [Commented] (SPARK-14082) Add support for GPU resource when running on Mesos

2016-05-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281371#comment-15281371 ] Kazuaki Ishizaki commented on SPARK-14082: -- Sounds great. We expect that our

[jira] [Updated] (SPARK-15117) Generate code that get a value in each compressed column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-15117: - Target Version/s: 2.1.0 > Generate code that get a value in each compressed column from

[jira] [Created] (SPARK-15117) Generate code that get a value in each compressed column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-15117: Summary: Generate code that get a value in each compressed column from CachedBatch when DataFrame.cache() is called Key: SPARK-15117 URL:

[jira] [Updated] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Description: When DataFrame.cache() is called, data is stored as column-oriented storage

[jira] [Updated] (SPARK-14098) Generate code that get a float/double value in each column from CachedBatch when DataFrame.cache() is called

2016-05-04 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Summary: Generate code that get a float/double value in each column from CachedBatch

[jira] [Updated] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-04-25 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14098: - Target Version/s: 2.0.0 > Generate code that get a value in each column from CachedBatch

[jira] [Commented] (SPARK-14138) Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

2016-04-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254061#comment-15254061 ] Kazuaki Ishizaki commented on SPARK-14138: -- Yes, it will be included in 1.6.2 and 2.0.0. >

[jira] [Commented] (SPARK-13904) Add support for pluggable cluster manager

2016-04-19 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247498#comment-15247498 ] Kazuaki Ishizaki commented on SPARK-13904: -- I agree with you since SPARK-14689 addresses. > Add

[jira] [Commented] (SPARK-13904) Add support for pluggable cluster manager

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244805#comment-15244805 ] Kazuaki Ishizaki commented on SPARK-13904: -- To merge this PR may have begun causing test

[jira] [Issue Comment Deleted] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14690: - Comment: was deleted (was: Add a link to the original JIRA) > [SQL] SPARK-8020 fails in

[jira] [Closed] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-14690. Add a link to the original JIRA > [SQL] SPARK-8020 fails in Jenkins for master >

[jira] [Commented] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15244798#comment-15244798 ] Kazuaki Ishizaki commented on SPARK-14690: -- I see. I will reopen the original JIRA soon. >

[jira] [Resolved] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-14690. -- Resolution: Duplicate > [SQL] SPARK-8020 fails in Jenkins for master >

[jira] [Updated] (SPARK-14690) [SQL] SPARK-8020 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-14690: - Summary: [SQL] SPARK-8020 fails in Jenkins for master (was: [SQL] SPARK-9757 fails in

[jira] [Created] (SPARK-14690) [SQL] SPARK-9757 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14690: Summary: [SQL] SPARK-9757 fails in Jenkins for master Key: SPARK-14690 URL: https://issues.apache.org/jira/browse/SPARK-14690 Project: Spark Issue

[jira] [Created] (SPARK-14689) [SQL] SPARK-9757 fails in Jenkins for master

2016-04-17 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14689: Summary: [SQL] SPARK-9757 fails in Jenkins for master Key: SPARK-14689 URL: https://issues.apache.org/jira/browse/SPARK-14689 Project: Spark Issue

[jira] [Commented] (SPARK-14656) Benchmark.getPorcessorName() always return "Unknown processor" on Linux

2016-04-15 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15242688#comment-15242688 ] Kazuaki Ishizaki commented on SPARK-14656: -- The root cause is described at

[jira] [Created] (SPARK-14656) Benchmark.getPorcessorName() always return "Unknown processor" on Linux

2016-04-15 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14656: Summary: Benchmark.getPorcessorName() always return "Unknown processor" on Linux Key: SPARK-14656 URL: https://issues.apache.org/jira/browse/SPARK-14656

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-04-01 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222634#comment-15222634 ] Kazuaki Ishizaki commented on SPARK-14083: -- Good point. On the other hand, should we follow JVM

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-03-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15220055#comment-15220055 ] Kazuaki Ishizaki commented on SPARK-14083: -- Thank you. Makes sense to me. It seems that this

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-03-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219637#comment-15219637 ] Kazuaki Ishizaki commented on SPARK-14083: -- Looks interesting. I am also interested in

[jira] [Commented] (SPARK-14083) Analyze JVM bytecode and turn closures into Catalyst expressions

2016-03-28 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213959#comment-15213959 ] Kazuaki Ishizaki commented on SPARK-14083: -- Another possible bytecode analysis library is WALA

[jira] [Commented] (SPARK-14138) Generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

2016-03-26 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213071#comment-15213071 ] Kazuaki Ishizaki commented on SPARK-14138: -- I will make a PR that includes two solutions: 1.

[jira] [Comment Edited] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-03-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209047#comment-15209047 ] Kazuaki Ishizaki edited comment on SPARK-14098 at 3/23/16 8:02 PM: ---

[jira] [Commented] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-03-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15209047#comment-15209047 ] Kazuaki Ishizaki commented on SPARK-14098: -- After fixing some trivial compilation failures, I

[jira] [Created] (SPARK-14098) Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called

2016-03-23 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14098: Summary: Generate code that get a value in each column from CachedBatch when DataFrame.cache() is called Key: SPARK-14098 URL:

[jira] [Created] (SPARK-14072) Show JVM information when we run Benchmark

2016-03-22 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-14072: Summary: Show JVM information when we run Benchmark Key: SPARK-14072 URL: https://issues.apache.org/jira/browse/SPARK-14072 Project: Spark Issue

[jira] [Created] (SPARK-13844) Generate better code for filters with a non-nullable column

2016-03-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13844: Summary: Generate better code for filters with a non-nullable column Key: SPARK-13844 URL: https://issues.apache.org/jira/browse/SPARK-13844 Project: Spark

[jira] [Created] (SPARK-13805) Direct consume ColumnVector in generated code when ColumnarBatch is used

2016-03-10 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13805: Summary: Direct consume ColumnVector in generated code when ColumnarBatch is used Key: SPARK-13805 URL: https://issues.apache.org/jira/browse/SPARK-13805

[jira] [Updated] (SPARK-13644) Add the source file name and line into Logger when an exception occurs in the generated code

2016-03-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-13644: - Summary: Add the source file name and line into Logger when an exception occurs in the

[jira] [Updated] (SPARK-13644) add the source file name and line into Logger when an exception occurs in the generated code

2016-03-03 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-13644: - Issue Type: Improvement (was: Bug) Summary: add the source file name and line

[jira] [Created] (SPARK-13644) [SQL] add the source file name and line into Logger when an exception occurs in the generated code

2016-03-03 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13644: Summary: [SQL] add the source file name and line into Logger when an exception occurs in the generated code Key: SPARK-13644 URL:

[jira] [Commented] (SPARK-13530) Add ShortType support to UnsafeRowParquetRecordReader

2016-02-27 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15170636#comment-15170636 ] Kazuaki Ishizaki commented on SPARK-13530: -- I saw the same exception in my environment. Thank

[jira] [Closed] (SPARK-12907) Use BitSet to represent null fields in ColumnVector

2016-02-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki closed SPARK-12907. Resolution: Won't Fix > Use BitSet to represent null fields in ColumnVector >

[jira] [Commented] (SPARK-13421) Make output of a SparkPlan configurable

2016-02-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15160157#comment-15160157 ] Kazuaki Ishizaki commented on SPARK-13421: -- I am willing to work for and contribute to this. Are

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-23 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159649#comment-15159649 ] Kazuaki Ishizaki commented on SPARK-13431: -- I identified why this problem occurs only in maven.

[jira] [Created] (SPARK-13437) Add InternalColumn

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13437: Summary: Add InternalColumn Key: SPARK-13437 URL: https://issues.apache.org/jira/browse/SPARK-13437 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157297#comment-15157297 ] Kazuaki Ishizaki edited comment on SPARK-13431 at 2/22/16 5:17 PM: --- The

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157297#comment-15157297 ] Kazuaki Ishizaki commented on SPARK-13431: -- The size of a static initializer method of

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157208#comment-15157208 ] Kazuaki Ishizaki commented on SPARK-13431: -- I am using mvn, and executed the following command

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157182#comment-15157182 ] Kazuaki Ishizaki commented on SPARK-13431: -- I succeeded to build Spark by commenting out lines

[jira] [Commented] (SPARK-13431) Maven build fails due to: Method code too large! in Catalyst

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15157104#comment-15157104 ] Kazuaki Ishizaki commented on SPARK-13431: -- According to AMPLAB's Jenkins, the first failure

[jira] [Created] (SPARK-13432) Add the origin of the source code into a generated Java code

2016-02-22 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-13432: Summary: Add the origin of the source code into a generated Java code Key: SPARK-13432 URL: https://issues.apache.org/jira/browse/SPARK-13432 Project: Spark

[jira] [Created] (SPARK-12907) Use BitSet to represent null fields in ColumnVector

2016-01-19 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-12907: Summary: Use BitSet to represent null fields in ColumnVector Key: SPARK-12907 URL: https://issues.apache.org/jira/browse/SPARK-12907 Project: Spark

[jira] [Commented] (SPARK-3785) Support off-loading computations to a GPU

2016-01-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082859#comment-15082859 ] Kazuaki Ishizaki commented on SPARK-3785: - To use CUDA is an intermediate approach to evaluate

[jira] [Commented] (SPARK-3785) Support off-loading computations to a GPU

2016-01-05 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15082881#comment-15082881 ] Kazuaki Ishizaki commented on SPARK-3785: - # You can specify cpu-cores by using conventional Spark

<    2   3   4   5   6   7   8   >