[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate

2020-01-08 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate URL: https://github.com/apache/spark/pull/24637#issuecomment-572144816 Could you file a JIRA for that, @cloud-fan ?

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate

2020-01-08 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate URL: https://github.com/apache/spark/pull/24637#issuecomment-572144426 Thank you for reporting, @cloud-fan ! This is

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate

2019-07-18 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate URL: https://github.com/apache/spark/pull/24637#issuecomment-512872770 Thank you for adding more. Then, please do a white list approach for those four expressions as I mentioned here.

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate

2019-07-17 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate URL: https://github.com/apache/spark/pull/24637#issuecomment-512358172 Thank you for adding a new test. Are you going to add more tests since `Stack` is one of them? In fact, we need

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate

2019-07-12 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate URL: https://github.com/apache/spark/pull/24637#issuecomment-510760456 Thank you so much, @viirya ! This is an

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode

2019-07-10 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510336289 I made a PR to you which reverts all the other benchmark result to the master and adds the newly

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode

2019-07-10 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510330614 Weirdly. The benchmark suite failed with the exact same reason at the first run. I'm running the

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode

2019-07-10 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510287567 Sure. I'll rerun and make a PR to you tonight.

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode

2019-07-10 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510266629 This is the result on this PR -

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode

2019-07-10 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-510253829 Oh, I faced the following during benchmark. It's weird. Let me rerun it. ``` [info] 22:18:13.279

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode

2019-06-18 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-503380719 Retest this please.

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode

2019-06-15 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-502417605 Retest this please.

[GitHub] [spark] dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode

2019-05-19 Thread GitBox
dongjoon-hyun commented on issue #24637: [SPARK-27707][SQL] Prune unnecessary nested fields from Generate to address performance issue in explode URL: https://github.com/apache/spark/pull/24637#issuecomment-493838695 cc @dbtsai