[ 
https://issues.apache.org/jira/browse/HIVE-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-11394:
--------------------------------
    Description: 
Add detail to the EXPLAIN output showing why a Map and Reduce work is not 
vectorized.

New syntax is: EXPLAIN VECTORIZATION \[ONLY\] 
\[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]

The ONLY option suppresses most non-vectorization elements.

SUMMARY shows vectorization information for the PLAN (is vectorization enabled) 
and a summary of Map and Reduce work.

OPERATOR shows vectorization information for operators.  E.g. Filter 
Vectorization.  It includes all information of SUMMARY, too.

EXPRESSION shows vectorization information for expressions.  E.g. 
predicateExpression.  It includes all information of SUMMARY and OPERATOR, too.

DETAIL shows very vectorization information.
It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.

The optional clause defaults are not ONLY and SUMMARY.

Here are some examples:

EXPLAIN VECTORIZATION example:

(Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization sections)

Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION SUMMARY.

{code}
coming soon…
{code}


EXPLAIN VECTORIZATION OPERATOR

Notice the added  Select Vectorization, Group By Vectorization, Reduce Sink 
Vectorization sections in this example.

{code}
coming soon…
{code}

EXPLAIN VECTORIZATION EXPRESSION

Notice the aaaaa in this example.

{code}
coming soon…
{code}


EXPLAIN VECTORIZATION DETAIL

Notice the aaaaa in this example.

{code}
coming soon…
{code}


EXPLAIN VECTORIZATION ONLY example:

{code}
coming soon…
{code}

EXPLAIN VECTORIZATION ONLY OPERATOR example:

{code}
coming soon…
{code}

EXPLAIN VECTORIZATION ONLY EXPRESSION example:

{code}
{code}

EXPLAIN VECTORIZATION ONLY DETAIL example:

{code}
coming soon…
{code}


The standard @Explain Annotation Type is used.  A new 'vectorization' 
annotation marks each new class and method.

Works for FORMATTED, like other non-vectorization EXPLAIN variations.

EXPLAIN VECTORIZATION FORMATTED example:

{code}
coming soon…
{code}

or pretty printed:

{code}
coming soon…
{code}


  was:
Add detail to the EXPLAIN output showing why a Map and Reduce work is not 
vectorized.

New syntax is: EXPLAIN VECTORIZATION \[ONLY\] \[SUMMARY|DETAIL\]

The ONLY option suppresses most non-vectorization elements.

SUMMARY shows vectorization information for the PLAN (is vectorization enabled) 
and a summary of Map and Reduce work.

The optional clause defaults are not ONLY and SUMMARY.

Here are some examples:

EXPLAIN VECTORIZATION example:

(Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization sections)

It is the same as EXPLAIN VECTORIZATION SUMMARY.

{code}
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
    Tez
…
      Edges:
        Reducer 2 <- Map 1 (SIMPLE_EDGE)
…
      Vertices:
        Map 1 
            Map Operator Tree:
                TableScan
                  alias: decimal_date_test
                  Statistics: Num rows: 12288 Data size: 2467616 Basic stats: 
COMPLETE Column stats: NONE
                  Filter Operator
                    predicate: cdate BETWEEN 1969-12-30 AND 1970-01-02 (type: 
boolean)
                    Statistics: Num rows: 6144 Data size: 1233808 Basic stats: 
COMPLETE Column stats: NONE
                    Select Operator
                      expressions: cdate (type: date)
                      outputColumnNames: _col0
                      Statistics: Num rows: 6144 Data size: 1233808 Basic 
stats: COMPLETE Column stats: NONE
                      Reduce Output Operator
                        key expressions: _col0 (type: date)
                        sort order: +
                        Statistics: Num rows: 6144 Data size: 1233808 Basic 
stats: COMPLETE Column stats: NONE
            Execution mode: vectorized, llap
            LLAP IO: all inputs
            Map Vectorization:
                enabled: true
                enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
                groupByVectorOutput: true
                inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
                allNative: false
                usesVectorUDFAdaptor: false
                vectorized: true
        Reducer 2 
            Execution mode: vectorized, llap
            Reduce Vectorization:
                enabled: true
                enableConditionsMet: hive.vectorized.execution.reduce.enabled 
IS true, hive.execution.engine tez IN [tez, spark] IS true
                groupByVectorOutput: true
                allNative: false
                usesVectorUDFAdaptor: false
                vectorized: true
            Reduce Operator Tree:
              Select Operator
                expressions: KEY.reducesinkkey0 (type: date)
                outputColumnNames: _col0
                Statistics: Num rows: 6144 Data size: 1233808 Basic stats: 
COMPLETE Column stats: NONE
                File Output Operator
                  compressed: false
                  Statistics: Num rows: 6144 Data size: 1233808 Basic stats: 
COMPLETE Column stats: NONE
                  table:
                      input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
                      output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

{code}


EXPLAIN VECTORIZATION DETAIL

(Note the added  Select Vectorization, Group By Vectorization, Reduce Sink 
Vectorization sections in this example)

{code}
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
    Tez
…
      Edges:
        Reducer 2 <- Map 1 (SIMPLE_EDGE)
        Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
…
      Vertices:
        Map 1 
            Map Operator Tree:
                TableScan
                  alias: vectortab2korc
                  Statistics: Num rows: 2000 Data size: 918712 Basic stats: 
COMPLETE Column stats: NONE
                  Select Operator
                    expressions: bo (type: boolean), b (type: bigint)
                    outputColumnNames: bo, b
                    Select Vectorization:
                        className: VectorSelectOperator
                        native: true
                        nativeConditionsMet: Supported IS true
                        selectExpressions: IdentityExpression[7:boolean], 
IdentityExpression[3:bigint]
                        vectorized: true
                    Statistics: Num rows: 2000 Data size: 918712 Basic stats: 
COMPLETE Column stats: NONE
                    Group By Operator
                      aggregations: max(b)
                      Group By Vectorization:
                          aggregators: 
VectorUDAFMaxLong(IdentityExpression[3:bigint])
                          className: VectorGroupByOperator
                          vectorOutput: true
                          keyExpressions: IdentityExpression[7:boolean]
                          native: false
                          nativeConditionsNotMet: Supported IS false
                          vectorized: true
                      keys: bo (type: boolean)
                      mode: hash
                      outputColumnNames: _col0, _col1
                      Statistics: Num rows: 2000 Data size: 918712 Basic stats: 
COMPLETE Column stats: NONE
                      Reduce Output Operator
                        key expressions: _col0 (type: boolean)
                        sort order: +
                        Map-reduce partition columns: _col0 (type: boolean)
                        Reduce Sink Vectorization:
                            className: VectorReduceSinkLongOperator
                            native: true
                            nativeConditionsMet: 
hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine 
tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS 
true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, 
BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
                            vectorized: true
                        Statistics: Num rows: 2000 Data size: 918712 Basic 
stats: COMPLETE Column stats: NONE
                        value expressions: _col1 (type: bigint)
            Execution mode: vectorized
            Map Vectorization:
                enabled: true
                enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
                groupByVectorOutput: true
                inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
                allNative: false
                usesVectorUDFAdaptor: false
                vectorized: true
        Reducer 2 
            Execution mode: vectorized
            Reduce Vectorization:
                enabled: true
                enableConditionsMet: hive.vectorized.execution.reduce.enabled 
IS true, hive.execution.engine tez IN [tez, spark] IS true
                groupByVectorOutput: true
                allNative: false
                usesVectorUDFAdaptor: false
                vectorized: true
            Reduce Operator Tree:
              Group By Operator
                aggregations: max(VALUE._col0)
                Group By Vectorization:
                    aggregators: VectorUDAFMaxLong(IdentityExpression[1:bigint])
                    className: VectorGroupByOperator
                    vectorOutput: true
                    keyExpressions: IdentityExpression[0:boolean]
                    native: false
                    nativeConditionsNotMet: Supported IS false
                    vectorized: true
                keys: KEY._col0 (type: boolean)
                mode: mergepartial
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 1000 Data size: 459356 Basic stats: 
COMPLETE Column stats: NONE
                Reduce Output Operator
                  key expressions: _col0 (type: boolean)
                  sort order: -
                  Reduce Sink Vectorization:
                      className: VectorReduceSinkOperator
                      native: false
                      nativeConditionsMet: 
hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine 
tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS 
true, No TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for 
keys IS true, LazyBinarySerDe for values IS true
                      nativeConditionsNotMet: Uniform Hash IS false
                      vectorized: true
                  Statistics: Num rows: 1000 Data size: 459356 Basic stats: 
COMPLETE Column stats: NONE
                  value expressions: _col1 (type: bigint)
…
{code}


EXPLAIN VECTORIZATION ONLY example:

{code}
PLAN VECTORIZATION:
  enabled: true
  enabledConditionsMet: [hive.vectorized.execution.enabled IS true]

STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
    Tez
      Edges:
        Map 1 <- Map 2 (BROADCAST_EDGE)
      Vertices:
        Map 1 
            Map Vectorization:
                enabled: true
                enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
                groupByVectorOutput: true
                inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
                allNative: false
                usesVectorUDFAdaptor: false
                vectorized: true
        Map 2 
            Map Vectorization:
                enabled: true
                enabledConditionsMet: 
hive.vectorized.use.vectorized.input.format IS true
                groupByVectorOutput: true
                inputFileFormats: 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
                allNative: true
                usesVectorUDFAdaptor: false
                vectorized: true

  Stage: Stage-0
{code}


The standard @Explain Annotation Type is used.  A new 'vectorization' 
annotation marks each new class and method.

Works for FORMATTED, like other non-vectorization EXPLAIN variations.

EXPLAIN VECTORIZATION ONLY SUMMARY FORMATTED

{code}
{"PLAN 
VECTORIZATION":{"enabled":true,"enabledConditionsMet":["hive.vectorized.execution.enabled
 IS true"]},"STAGE DEPENDENCIES":{"Stage-1":{"ROOT 
STAGE":"TRUE"},"Stage-0":{"DEPENDENT STAGES":"Stage-1"}},"STAGE 
PLANS":{"Stage-1":{"Tez":{"Edges:":{"Map 1":[{"parent":"Map 
3","type":"BROADCAST_EDGE"},{"parent":"Map 
4","type":"BROADCAST_EDGE"}],"Reducer 2":{"parent":"Map 
1","type":"SIMPLE_EDGE"}},"Vertices:":{"Map 1":{"Map 
Vectorization:":{"enabled:":"true","enabledConditionsMet:":["hive.vectorized.use.vectorized.input.format
 IS 
true"],"groupByVectorOutput:":"true","inputFileFormats:":["org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"],"allNative:":"false","usesVectorUDFAdaptor:":"false","vectorized:":"true"}},"Map
 3":{"Map 
Vectorization:":{"enabled:":"true","enabledConditionsMet:":["hive.vectorized.use.vectorized.input.format
 IS 
true"],"groupByVectorOutput:":"true","inputFileFormats:":["org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"],"allNative:":"true","usesVectorUDFAdaptor:":"false","vectorized:":"true"}},"Map
 4":{"Map 
Vectorization:":{"enabled:":"true","enabledConditionsMet:":["hive.vectorized.use.vectorized.input.format
 IS 
true"],"groupByVectorOutput:":"true","inputFileFormats:":["org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"],"allNative:":"true","usesVectorUDFAdaptor:":"false","vectorized:":"true"}},"Reducer
 2":{"Reduce 
Vectorization:":{"enabled:":"true","enableConditionsMet:":["hive.vectorized.execution.reduce.enabled
 IS true","hive.execution.engine tez IN [tez, spark] IS 
true"],"groupByVectorOutput:":"true","allNative:":"false","usesVectorUDFAdaptor:":"false","vectorized:":"true"}}}}},"Stage-0":{}}}
{code}

or pretty printed:

{code}
{
  "PLAN VECTORIZATION": {
    "enabled": true,
    "enabledConditionsMet": [
      "hive.vectorized.execution.enabled IS true"
    ]
  },
  "STAGE DEPENDENCIES": {
    "Stage-1": {
      "ROOT STAGE": "TRUE"
    },
    "Stage-0": {
      "DEPENDENT STAGES": "Stage-1"
    }
  },
  "STAGE PLANS": {
    "Stage-1": {
      "Tez": {
        "Edges:": {
          "Map 1": [
            {
              "parent": "Map 3",
              "type": "BROADCAST_EDGE"
            },
            {
              "parent": "Map 4",
              "type": "BROADCAST_EDGE"
            }
          ],
          "Reducer 2": {
            "parent": "Map 1",
            "type": "SIMPLE_EDGE"
          }
        },
        "Vertices:": {
          "Map 1": {
            "Map Vectorization:": {
              "enabled:": "true",
              "enabledConditionsMet:": [
                "hive.vectorized.use.vectorized.input.format IS true"
              ],
              "groupByVectorOutput:": "true",
              "inputFileFormats:": [
                "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"
              ],
              "allNative:": "false",
              "usesVectorUDFAdaptor:": "false",
              "vectorized:": "true"
            }
          },
          "Map 3": {
            "Map Vectorization:": {
              "enabled:": "true",
              "enabledConditionsMet:": [
                "hive.vectorized.use.vectorized.input.format IS true"
              ],
              "groupByVectorOutput:": "true",
              "inputFileFormats:": [
                "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"
              ],
              "allNative:": "true",
              "usesVectorUDFAdaptor:": "false",
              "vectorized:": "true"
            }
          },
          "Map 4": {
            "Map Vectorization:": {
              "enabled:": "true",
              "enabledConditionsMet:": [
                "hive.vectorized.use.vectorized.input.format IS true"
              ],
              "groupByVectorOutput:": "true",
              "inputFileFormats:": [
                "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"
              ],
              "allNative:": "true",
              "usesVectorUDFAdaptor:": "false",
              "vectorized:": "true"
            }
          },
          "Reducer 2": {
            "Reduce Vectorization:": {
              "enabled:": "true",
              "enableConditionsMet:": [
                "hive.vectorized.execution.reduce.enabled IS true",
                "hive.execution.engine tez IN [tez, spark] IS true"
              ],
              "groupByVectorOutput:": "true",
              "allNative:": "false",
              "usesVectorUDFAdaptor:": "false",
              "vectorized:": "true"
            }
          }
        }
      }
    },
    "Stage-0": {
      
    }
{code}



> Enhance EXPLAIN display for vectorization
> -----------------------------------------
>
>                 Key: HIVE-11394
>                 URL: https://issues.apache.org/jira/browse/HIVE-11394
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>            Reporter: Matt McCline
>            Assignee: Matt McCline
>            Priority: Critical
>         Attachments: HIVE-11394.01.patch, HIVE-11394.02.patch, 
> HIVE-11394.03.patch, HIVE-11394.04.patch, HIVE-11394.05.patch, 
> HIVE-11394.06.patch, HIVE-11394.07.patch
>
>
> Add detail to the EXPLAIN output showing why a Map and Reduce work is not 
> vectorized.
> New syntax is: EXPLAIN VECTORIZATION \[ONLY\] 
> \[SUMMARY|OPERATOR|EXPRESSION|DETAIL\]
> The ONLY option suppresses most non-vectorization elements.
> SUMMARY shows vectorization information for the PLAN (is vectorization 
> enabled) and a summary of Map and Reduce work.
> OPERATOR shows vectorization information for operators.  E.g. Filter 
> Vectorization.  It includes all information of SUMMARY, too.
> EXPRESSION shows vectorization information for expressions.  E.g. 
> predicateExpression.  It includes all information of SUMMARY and OPERATOR, 
> too.
> DETAIL shows very vectorization information.
> It includes all information of SUMMARY, OPERATOR, and EXPRESSION too.
> The optional clause defaults are not ONLY and SUMMARY.
> Here are some examples:
> EXPLAIN VECTORIZATION example:
> (Note the PLAN VECTORIZATION, Map Vectorization, Reduce Vectorization 
> sections)
> Since SUMMARY is the default, it is the output of EXPLAIN VECTORIZATION 
> SUMMARY.
> {code}
> coming soon…
> {code}
> EXPLAIN VECTORIZATION OPERATOR
> Notice the added  Select Vectorization, Group By Vectorization, Reduce Sink 
> Vectorization sections in this example.
> {code}
> coming soon…
> {code}
> EXPLAIN VECTORIZATION EXPRESSION
> Notice the aaaaa in this example.
> {code}
> coming soon…
> {code}
> EXPLAIN VECTORIZATION DETAIL
> Notice the aaaaa in this example.
> {code}
> coming soon…
> {code}
> EXPLAIN VECTORIZATION ONLY example:
> {code}
> coming soon…
> {code}
> EXPLAIN VECTORIZATION ONLY OPERATOR example:
> {code}
> coming soon…
> {code}
> EXPLAIN VECTORIZATION ONLY EXPRESSION example:
> {code}
> {code}
> EXPLAIN VECTORIZATION ONLY DETAIL example:
> {code}
> coming soon…
> {code}
> The standard @Explain Annotation Type is used.  A new 'vectorization' 
> annotation marks each new class and method.
> Works for FORMATTED, like other non-vectorization EXPLAIN variations.
> EXPLAIN VECTORIZATION FORMATTED example:
> {code}
> coming soon…
> {code}
> or pretty printed:
> {code}
> coming soon…
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to