[
https://issues.apache.org/jira/browse/HIVE-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15936330#comment-15936330
]
Hive QA commented on HIVE-16219:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859906/HIVE-16219.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4291/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4291/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4291/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-22 13:39:05.460
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4291/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-22 13:39:05.463
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at ce695b5 HIVE-15784: Vectorization: Turn on text vectorization by
default (vector serde) (Matt McCline, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at ce695b5 HIVE-15784: Vectorization: Turn on text vectorization by
default (vector serde) (Matt McCline, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-22 13:39:06.494
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OctetLength.java:1
error:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/OctetLength.java:
patch does not apply
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBinarySetFunctions.java:1
error:
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBinarySetFunctions.java:
patch does not apply
error: patch failed:
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDAFBinarySetFunctions.java:1
error:
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDAFBinarySetFunctions.java:
patch does not apply
error: patch failed:
ql/src/test/queries/clientpositive/udaf_binarysetfunctions.q:1
error: ql/src/test/queries/clientpositive/udaf_binarysetfunctions.q: patch does
not apply
error: patch failed:
ql/src/test/queries/clientpositive/vector_udf_character_length.q:1
error: ql/src/test/queries/clientpositive/vector_udf_character_length.q: patch
does not apply
error: patch failed:
ql/src/test/queries/clientpositive/vector_udf_octet_length.q:1
error: ql/src/test/queries/clientpositive/vector_udf_octet_length.q: patch does
not apply
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_udf_character_length.q.out:1
error:
ql/src/test/results/clientpositive/llap/vector_udf_character_length.q.out:
patch does not apply
error: patch failed:
ql/src/test/results/clientpositive/llap/vector_udf_octet_length.q.out:1
error: ql/src/test/results/clientpositive/llap/vector_udf_octet_length.q.out:
patch does not apply
error: patch failed:
ql/src/test/results/clientpositive/udaf_binarysetfunctions.q.out:1
error: ql/src/test/results/clientpositive/udaf_binarysetfunctions.q.out: patch
does not apply
error: patch failed:
ql/src/test/results/clientpositive/vector_udf_character_length.q.out:1
error: ql/src/test/results/clientpositive/vector_udf_character_length.q.out:
patch does not apply
error: patch failed:
ql/src/test/results/clientpositive/vector_udf_octet_length.q.out:1
error: ql/src/test/results/clientpositive/vector_udf_octet_length.q.out: patch
does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12859906 - PreCommit-HIVE-Build
> metastore notification_log contains serialized message with non functional
> fields
> ----------------------------------------------------------------------------------
>
> Key: HIVE-16219
> URL: https://issues.apache.org/jira/browse/HIVE-16219
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 2.2.0
> Reporter: anishek
> Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16219.1.patch, HIVE-16219.1.patch,
> HIVE-16219.2.patch
>
>
> the event notification logs stored in hive metastore have json serialized
> messages stored in NOTIFICATION_LOG table, these messages also store the
> serialized Thrift API objects in them. when doing a reply dump we are however
> serializing both the metadata for replication event + event Message +
> additional helper method getters representing the thrift objects.
> We should only serialize metadata for replication event + event Message
> for ex for create table :
> {code}
> {
> "eventType": "CREATE_TABLE",
> "server": "",
> "servicePrincipal": "",
> "db": "default",
> "table": "a",
> "tableObjJson":
> "{\"1\":{\"str\":\"a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1489552350},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"file:/tmp/warehouse/a\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.TextInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":-1},\"7\":{\"rec\":{\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",2,{\"field.delim\":\"\\n\",\"serialization.format\":\"\\n\"}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"totalSize\":\"0\",\"EXTERNAL\":\"TRUE\",\"numRows\":\"0\",\"rawDataSize\":\"0\",\"COLUMN_STATS_ACCURATE\":\"{\\\"BASIC_STATS\\\":\\\"true\\\"}\",\"numFiles\":\"0\",\"transient_lastDdlTime\":\"1489552350\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"13\":{\"rec\":{\"1\":{\"map\":[\"str\",\"lst\",1,{\"anagarwal\":[\"rec\",4,{\"1\":{\"str\":\"INSERT\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"SELECT\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"UPDATE\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"DELETE\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}}]}]}}},\"14\":{\"tf\":0}}",
> "timestamp": 1489552350,
> "files": [],
> "tableObj": {
> "tableName": "a",
> "dbName": "default",
> "owner": "anagarwal",
> "createTime": 1489552350,
> "lastAccessTime": 0,
> "retention": 0,
> "sd": {
> "cols": [
> {
> "name": "name",
> "type": "string",
> "comment": null,
> "setName": true,
> "setType": true,
> "setComment": false
> }
> ],
> "location": "file:/tmp/warehouse/a",
> "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
> "outputFormat":
> "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
> "compressed": false,
> "numBuckets": -1,
> "serdeInfo": {
> "name": null,
> "serializationLib":
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
> "parameters": {
> "serialization.format": "\n",
> "field.delim": "\n"
> },
> "setName": false,
> "parametersSize": 2,
> "setParameters": true,
> "setSerializationLib": true
> },
> "bucketCols": [],
> "sortCols": [],
> "parameters": {},
> "skewedInfo": {
> "skewedColNames": [],
> "skewedColValues": [],
> "skewedColValueLocationMaps": {},
> "setSkewedColNames": true,
> "setSkewedColValues": true,
> "setSkewedColValueLocationMaps": true,
> "skewedColNamesSize": 0,
> "skewedColNamesIterator": [],
> "skewedColValuesSize": 0,
> "skewedColValuesIterator": [],
> "skewedColValueLocationMapsSize": 0
> },
> "storedAsSubDirectories": false,
> "setSkewedInfo": true,
> "parametersSize": 0,
> "colsSize": 1,
> "setParameters": true,
> "setLocation": true,
> "setInputFormat": true,
> "setCols": true,
> "setOutputFormat": true,
> "setSerdeInfo": true,
> "setBucketCols": true,
> "setSortCols": true,
> "colsIterator": [
> {
> "name": "name",
> "type": "string",
> "comment": null,
> "setName": true,
> "setType": true,
> "setComment": false
> }
> ],
> "bucketColsSize": 0,
> "bucketColsIterator": [],
> "sortColsSize": 0,
> "sortColsIterator": [],
> "setStoredAsSubDirectories": true,
> "setCompressed": true,
> "setNumBuckets": true
> },
> "partitionKeys": [],
> "parameters": {
> "totalSize": "0",
> "EXTERNAL": "TRUE",
> "numRows": "0",
> "rawDataSize": "0",
> "COLUMN_STATS_ACCURATE": "{\"BASIC_STATS\":\"true\"}",
> "numFiles": "0",
> "transient_lastDdlTime": "1489552350"
> },
> "viewOriginalText": null,
> "viewExpandedText": null,
> "tableType": "EXTERNAL_TABLE",
> "privileges": {
> "userPrivileges": {
> "anagarwal": [
> {
> "privilege": "INSERT",
> "createTime": -1,
> "grantor": "anagarwal",
> "grantorType": "USER",
> "grantOption": true,
> "setCreateTime": true,
> "setGrantOption": true,
> "setPrivilege": true,
> "setGrantor": true,
> "setGrantorType": true
> },
> {
> "privilege": "SELECT",
> "createTime": -1,
> "grantor": "anagarwal",
> "grantorType": "USER",
> "grantOption": true,
> "setCreateTime": true,
> "setGrantOption": true,
> "setPrivilege": true,
> "setGrantor": true,
> "setGrantorType": true
> },
> {
> "privilege": "UPDATE",
> "createTime": -1,
> "grantor": "anagarwal",
> "grantorType": "USER",
> "grantOption": true,
> "setCreateTime": true,
> "setGrantOption": true,
> "setPrivilege": true,
> "setGrantor": true,
> "setGrantorType": true
> },
> {
> "privilege": "DELETE",
> "createTime": -1,
> "grantor": "anagarwal",
> "grantorType": "USER",
> "grantOption": true,
> "setCreateTime": true,
> "setGrantOption": true,
> "setPrivilege": true,
> "setGrantor": true,
> "setGrantorType": true
> }
> ]
> },
> "groupPrivileges": null,
> "rolePrivileges": null,
> "rolePrivilegesSize": 0,
> "setUserPrivileges": true,
> "setGroupPrivileges": false,
> "setRolePrivileges": false,
> "userPrivilegesSize": 1,
> "groupPrivilegesSize": 0
> },
> "temporary": false,
> "rewriteEnabled": false,
> "setTableName": true,
> "setDbName": true,
> "setOwner": true,
> "setViewOriginalText": false,
> "setViewExpandedText": false,
> "setTableType": true,
> "setPrivileges": true,
> "setCreateTime": true,
> "setLastAccessTime": true,
> "setRetention": true,
> "partitionKeysIterator": [],
> "parametersSize": 7,
> "setTemporary": true,
> "setRewriteEnabled": false,
> "setParameters": true,
> "setPartitionKeys": true,
> "setSd": true,
> "partitionKeysSize": 0
> }
> }
> {code}
> it should only be the json message required as :
> {code}
> {
> "eventType": "CREATE_TABLE",
> "server": "",
> "servicePrincipal": "",
> "db": "default",
> "table": "a",
> "tableObjJson":
> "{\"1\":{\"str\":\"a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1489552350},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"file:/tmp/warehouse/a\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.TextInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":-1},\"7\":{\"rec\":{\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",2,{\"field.delim\":\"\\n\",\"serialization.format\":\"\\n\"}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"totalSize\":\"0\",\"EXTERNAL\":\"TRUE\",\"numRows\":\"0\",\"rawDataSize\":\"0\",\"COLUMN_STATS_ACCURATE\":\"{\\\"BASIC_STATS\\\":\\\"true\\\"}\",\"numFiles\":\"0\",\"transient_lastDdlTime\":\"1489552350\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"13\":{\"rec\":{\"1\":{\"map\":[\"str\",\"lst\",1,{\"anagarwal\":[\"rec\",4,{\"1\":{\"str\":\"INSERT\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"SELECT\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"UPDATE\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"DELETE\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}}]}]}}},\"14\":{\"tf\":0}}",
> "timestamp": 1489552350,
> "files": [],
> }
> {code}
> this will require adding serialization features to mapper use such that it
> only serializes the annotated fields.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
