[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947591#comment-15947591 ] Eugene Koifman commented on HIVE-15691: --- done https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest#StreamingDataIngest-StrictRegexWriter > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Fix For: 1.3.0, 1.2.2, 2.3.0, 3.0.0 > > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15946247#comment-15946247 ] Kalyan commented on HIVE-15691: --- Hi [~ekoifman], if possible please update the document with below text. StrictRegexWriter Class StrictRegexWriter implements the RecordWriter interface. It accepts input records, regex that in text format and writes them to Hive. It converts the text record using proper regex directly into an Object using RegexSerDe, which is then passed on to the underlying AcidOutputFormat's record updater for the appropriate bucket. See Javadoc. I am not getting write permission to update the docs. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Fix For: 1.3.0, 1.2.2, 2.3.0, 3.0.0 > > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945489#comment-15945489 ] Eugene Koifman commented on HIVE-15691: --- https://github.com/apache/hive/commit/a3f718f7f7b1d850e67a3019cbf99d9699e09d7d - branch-1.2 https://github.com/apache/hive/commit/ea3be9549dca7eaed5e838bbcb69d2372817ce42 - 2.3.0 and 3.0 branch-1 no longer compiles - there are multiple failures: HIVE-13840 and HIVE-14255 but maybe others > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Fix For: 1.2.2, 3.0.0, 2.3.0 > > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943717#comment-15943717 ] Eugene Koifman commented on HIVE-15691: --- +1 HIVE-15691.1-branch-1.patch HIVE-15691.1-branch-1.2.patch > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942180#comment-15942180 ] Lefty Leverenz commented on HIVE-15691: --- [~ekoifman], please update this issue's status and fix version. You committed it to master before branch-2 was created, so it'll be in 2.3.0 and 3.0.0 (commit ea3be9549dca7eaed5e838bbcb69d2372817ce42). > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942178#comment-15942178 ] Lefty Leverenz commented on HIVE-15691: --- [~kalyanhadoop], the wiki has a GUI for editing and writing new docs. For write access, you need a Confluence username as described here: * [About This Wiki -- How to get permission to edit | https://cwiki.apache.org/confluence/display/Hive/AboutThisWiki#AboutThisWiki-Howtogetpermissiontoedit] Once you have write access, you'll find an Edit link on each wiki page (top right) and a Create button in the top menu bar. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942176#comment-15942176
]
Hive QA commented on HIVE-15691:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860547/HIVE-15691.1-branch-1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 161 failed/errored test(s), 8085 tests
executed
*Failed tests:*
{noformat}
TestAcidOnTez - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=358)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=391)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=368)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=398)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=416)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed
out) (batchId=383)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=390)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=342)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=377)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=345)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=355)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=402)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=350)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=385)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=419)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=420)
TestHS2ClearDanglingScratchDir - did not produce a TEST-*.xml file (likely
timed out) (batchId=406)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=407)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=394)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=396)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=370)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=360)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=373)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=422)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=423)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=403)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=375)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=451)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=410)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=421)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=415)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=412)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=448)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=447)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=445)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=450)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=411)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=417)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=418)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=382)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942145#comment-15942145
]
Hive QA commented on HIVE-15691:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860547/HIVE-15691.1-branch-1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 161 failed/errored test(s), 8085 tests
executed
*Failed tests:*
{noformat}
TestAcidOnTez - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=358)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=391)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=368)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=398)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=416)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed
out) (batchId=383)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=390)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=342)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=377)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=345)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=355)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=402)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=350)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=385)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=419)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=420)
TestHS2ClearDanglingScratchDir - did not produce a TEST-*.xml file (likely
timed out) (batchId=406)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=407)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=394)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=396)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=370)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=360)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=373)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=422)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=423)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=403)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=375)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=451)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=410)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=421)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=415)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=412)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=448)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=447)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=445)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=450)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=411)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=417)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=418)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=382)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15942075#comment-15942075 ] Kalyan commented on HIVE-15691: --- Hi [~leftylev], where i can find wiki source code. can you please help on this. Thanks > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1-branch-1.2.patch, > HIVE-15691.1-branch-1.patch, HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941912#comment-15941912 ] Pengcheng Xiong commented on HIVE-15691: I am deferring this to Hive 3.0 as we are going to cut the first RC and it is not marked as blocker. Please feel free to commit to the branch if this can be resolved before the release. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938815#comment-15938815 ] Eugene Koifman commented on HIVE-15691: --- the 1.3 version looks fine but 1.2.2 should only have c'tor without StreamingConection because the super class https://github.com/apache/hive/blob/branch-1.2/hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java doesn't have matching ones. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15938255#comment-15938255
]
Kalyan commented on HIVE-15691:
---
Hi [~ekoifman], as per my understanding below constructors signature expecting.
If it is fine, i will work out, otherwise any changes please suggest.
Hive-1.2.2 constructors:
public StrictRegexWriter(HiveEndPoint endPoint)
public StrictRegexWriter(HiveEndPoint endPoint, HiveConf conf)
public StrictRegexWriter(String regex, HiveEndPoint endPoint, HiveConf conf)
public StrictRegexWriter(HiveEndPoint endPoint, StreamingConnection conn)
public StrictRegexWriter(HiveEndPoint endPoint, HiveConf conf,
StreamingConnection conn)
public StrictRegexWriter(String regex, HiveEndPoint endPoint, HiveConf conf,
StreamingConnection conn)
Hive-1.3.0 constructors:
/**
* @deprecated As of release 1.3/2.1. Replaced by {@link
#StrictRegexWriter(HiveEndPoint, StreamingConnection)}
*/
public StrictRegexWriter(HiveEndPoint endPoint)
/**
* @deprecated As of release 1.3/2.1. Replaced by {@link
#StrictRegexWriter(HiveEndPoint, HiveConf, StreamingConnection)}
*/
public StrictRegexWriter(HiveEndPoint endPoint, HiveConf conf)
/**
* @deprecated As of release 1.3/2.1. Replaced by {@link
#StrictRegexWriter(String, HiveEndPoint, HiveConf, StreamingConnection)}
*/
public StrictRegexWriter(String regex, HiveEndPoint endPoint, HiveConf conf)
public StrictRegexWriter(HiveEndPoint endPoint, StreamingConnection conn)
public StrictRegexWriter(HiveEndPoint endPoint, HiveConf conf,
StreamingConnection conn)
public StrictRegexWriter(String regex, HiveEndPoint endPoint, HiveConf conf,
StreamingConnection conn)
Thanks
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
> -
>
> Key: HIVE-15691
> URL: https://issues.apache.org/jira/browse/HIVE-15691
> Project: Hive
> Issue Type: New Feature
> Components: HCatalog, Transactions
>Reporter: Kalyan
>Assignee: Kalyan
>Priority: Critical
> Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch,
> HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch,
> HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch,
> HIVE-15691-updated.patch
>
>
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink.
> It is similar to StrictJsonWriter available in hive.
> Dependency is there in flume to commit.
> FLUME-3036 : Create a RegexSerializer for Hive Sink.
> Patch is available for Flume, Please verify the below link
> https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937863#comment-15937863 ] Lefty Leverenz commented on HIVE-15691: --- Should this be documented in the wiki? Also, the status needs to be updated. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937057#comment-15937057 ] Eugene Koifman commented on HIVE-15691: --- HIVE-15691.5.patch committed to master https://github.com/apache/hive/commit/ea3be9549dca7eaed5e838bbcb69d2372817ce42 > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936955#comment-15936955 ] Eugene Koifman commented on HIVE-15691: --- +1 for HIVE-15691.5.patch for Hive2 [~kalyanhadoop] we still have an issue with Hive1 patches. (my bad). If you only want to check in to Hive1 master (future 1.3), then it's fine to have only the c'tors that have StreamingConnection param. If you want to check in to commit to both Hive1 1.2.2 and master, then we need c'tor w/o StreamingConnection for 1.2.2 and both versions for master (on master the ones w/o StreamingConnection should be deprecated). Otherwise someone may start using StrictRegexWriter on 1.2.2 with old style c'tors and when they upgrade to 1.3 it won't compile any more with the patch as is. I'll commit to Hive2 later today. Let me know what you want to do for Hive1. Also, it seems like the build bot only picks up 1 patch at a time, so if you submit both branch-1 and branch-1.2 at the same time only one of them is being run through the test > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936890#comment-15936890 ] Vaibhav Gumashta commented on HIVE-15691: - [~ekoifman] Please ignore the above comment: looks like this was a branch-1 run. The latest patch will need another run on 1.2. Thanks. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch, > HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936889#comment-15936889
]
Vaibhav Gumashta commented on HIVE-15691:
-
[~ekoifman] For the hive-1.2 branch, there are some extra test failures:
{code}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_mapjoin (batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_database_removes_partition_dirs
(batchId=98)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_table_removes_partition_dirs
(batchId=132)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization
(batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby10 (batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby11 (batchId=111)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby8 (batchId=114)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin5 (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_gb1
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_string_decimal
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing (batchId=82)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_drop_partition
(batchId=437)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_groupby2
(batchId=180)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_stats_only_null
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_union_fast_stats
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
(batchId=175)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_groupby2
(batchId=143)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap_auto
(batchId=138)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_ql_rewrite_gbtoidx_cbo_2
(batchId=147)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_right_side_join
(batchId=153)
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenFailure
(batchId=589)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=269)
{code}
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
> -
>
> Key: HIVE-15691
> URL: https://issues.apache.org/jira/browse/HIVE-15691
> Project: Hive
> Issue Type: New Feature
> Components: HCatalog, Transactions
>Reporter: Kalyan
>Assignee: Kalyan
>Priority: Critical
> Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch,
> HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691.5.patch,
> HIVE-15691-branch-1.2.patch, HIVE-15691-branch-1.patch, HIVE-15691.patch,
> HIVE-15691-updated.patch
>
>
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink.
> It is similar to StrictJsonWriter available in hive.
> Dependency is there in flume to commit.
> FLUME-3036 : Create a RegexSerializer for Hive Sink.
> Patch is available for Flume, Please verify the below link
> https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935690#comment-15935690
]
Hive QA commented on HIVE-15691:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859814/HIVE-15691-branch-1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 161 failed/errored test(s), 8085 tests
executed
*Failed tests:*
{noformat}
TestAcidOnTez - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=358)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=391)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=368)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=398)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=416)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed
out) (batchId=383)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=390)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=342)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=377)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=345)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=355)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=402)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=350)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=385)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=419)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=420)
TestHS2ClearDanglingScratchDir - did not produce a TEST-*.xml file (likely
timed out) (batchId=406)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=407)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=394)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=396)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=370)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=360)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=373)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=422)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=423)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=403)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=375)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=451)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=410)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=421)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=415)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=412)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=448)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=447)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=445)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=450)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=411)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=417)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=418)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=382)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935535#comment-15935535 ] Kalyan commented on HIVE-15691: --- Hi [~ekoifman], now i got the problem. given the patches as per comments. updated new patch (HIVE-15691-branch-1.patch) on branch-1 updated new patch (HIVE-15691.4.patch) on master Thanks > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691.4.patch, HIVE-15691-branch-1.2.patch, > HIVE-15691-branch-1.patch, HIVE-15691.patch, HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934983#comment-15934983 ] Eugene Koifman commented on HIVE-15691: --- The link above shows the signatures with StreamingConnection param in master branch of Hive1. This is where Hive 1.3 will be cut from so to be included in the future of 1.x line releases there must be signatures with StreamingConnection. There is no need to provide the deprecated version of constructors w/o StreamingConnection parameters since StrictRegexWriter is new API. 1.2 version (Hive1) doesn't have StreamingConnection parameter - it was introduced in HIVE-14114. If you'd like these changes to show up in upcoming 1.2.2 release then HIVE-15691-branch-1.2.patch is fine but we cannot check this into 1.2 w/o checking into 1.3 which requires a patch with appropriate signatures. HIVE-15691.3.patch looks ok for Hive2 but I don't see the build bot run for it. Perhaps there was some glitch in the system and it was missed. In such cases we usually resubmit the same patch but with new name (.4) to trigger another run. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934751#comment-15934751 ] Kalyan commented on HIVE-15691: --- In Hive1 current Writers like StrictJsonWriter and DelimitedInputWriter, there are still with out StreamingConnection parameters In Hive2 current Writers like StrictJsonWriter and DelimitedInputWriter, there are with StreamingConnection parameters i followed same implementation .. if required means better to change all the places also. may be better person to answer this [~roshan_naik] > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934738#comment-15934738 ] Eugene Koifman commented on HIVE-15691: --- The link I provided is for Hive1 - it's to prove that we do have StreamingConnection parameters in Hive1 master. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934724#comment-15934724 ] Kalyan commented on HIVE-15691: --- Hi [~ekoifman], as per your previous comments ... i removed those constructors in hive-2.x. if we want those constructors means use HIVE-15691.1.patch instead of HIVE-15691.3.patch for hive-2.x > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934699#comment-15934699 ] Eugene Koifman commented on HIVE-15691: --- https://github.com/apache/hive/blob/master/hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/StrictJsonWriter.java#L48-L83 > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan >Priority: Critical > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934100#comment-15934100
]
Vaibhav Gumashta commented on HIVE-15691:
-
[~ekoifman] I haven't created a release tag yet, so to get it into 1.2.2, it
should be good to commit it to branch-1.2. I looked at the test failures (from
the tests that ran and not timed out):
{code}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_from_utc_timestamp
(batchId=120)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_to_utc_timestamp
(batchId=33)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_bucket_map_join_tez1
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_smb_empty
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14
(batchId=164)
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_exchgpartition2lel
(batchId=136)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
(batchId=144)
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
(batchId=427)
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
(batchId=427)
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin (batchId=424)
{code}
All of these are flaky tests / need updated diff files which I have analyzed in
HIVE-15007, so the tests for this patch look good. However, if the patch is
revised, it will be good to get another run.
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
> -
>
> Key: HIVE-15691
> URL: https://issues.apache.org/jira/browse/HIVE-15691
> Project: Hive
> Issue Type: New Feature
> Components: HCatalog, Transactions
>Reporter: Kalyan
>Assignee: Kalyan
> Fix For: 1.3.0, 1.2.2, 2.2.0
>
> Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch,
> HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch,
> HIVE-15691-updated.patch
>
>
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink.
> It is similar to StrictJsonWriter available in hive.
> Dependency is there in flume to commit.
> FLUME-3036 : Create a RegexSerializer for Hive Sink.
> Patch is available for Flume, Please verify the below link
> https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934094#comment-15934094
]
Hive QA commented on HIVE-15691:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12859678/HIVE-15691-branch-1.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 133 failed/errored test(s), 7900 tests
executed
*Failed tests:*
{noformat}
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=340)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=371)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=350)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=356)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=394)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=370)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=360)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=359)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=325)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=358)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=328)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=337)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=382)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=332)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=365)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=397)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=398)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=386)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=373)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=352)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=342)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=355)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=400)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=401)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=383)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=357)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=429)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=388)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=390)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=426)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=425)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=423)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=428)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=389)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=395)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=396)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=363)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=361)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out)
(batchId=349)
TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out)
(batchId=353)
TestMetaStoreAuthorization - did not produce a TEST-*.xml file (likely timed
out)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933992#comment-15933992 ] Eugene Koifman commented on HIVE-15691: --- [~kalyanhadoop], sorry if I wasn't clear. Hive1 patch should have both versions of c'tors. The ones w/o Streaming connection should be deprecated. Basically it should mirror the other Streaming Writers. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Fix For: 1.3.0, 1.2.2, 2.2.0 > > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933990#comment-15933990 ] Eugene Koifman commented on HIVE-15691: --- [~vgumashta] I'd like to try to get this into 1.2.2. Technically this is not a bug fix so I don't know what rules we have wrt this > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Fix For: 1.3.0, 1.2.2, 2.2.0 > > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.3.patch, HIVE-15691-branch-1.2.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929219#comment-15929219 ] Eugene Koifman commented on HIVE-15691: --- [~kalyanhadoop], 1. can please follow https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-CreatingaPatch to name your patches 2. I don't see the updates in .1.patch to remote the deprecated constructors > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.patch, HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929039#comment-15929039 ] Kalyan commented on HIVE-15691: --- Hi [~ekoifman], [~roshan_naik] can you please take a look on above patch. Thanks > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.2.patch, > HIVE-15691.patch, HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923493#comment-15923493
]
Hive QA commented on HIVE-15691:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12858577/HIVE-15691.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4115/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4115/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4115/
Messages:
{noformat}
This message was trimmed, see log for full details
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/String.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/http/HttpStatus.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/HashMap.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/MediaType.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-core/1.14/jersey-core-1.14.jar(javax/ws/rs/core/Response.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar(org/codehaus/jackson/map/ObjectMapper.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Exception.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Throwable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Serializable.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Enum.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/Comparable.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-server/1.14/jersey-server-1.14.jar(com/sun/jersey/api/core/PackagesResourceConfig.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/com/sun/jersey/jersey-servlet/1.14/jersey-servlet-1.14.jar(com/sun/jersey/spi/container/servlet/ServletContainer.class)]]
[loading
ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/FileInputStream.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT.jar(org/apache/commons/lang3/StringUtils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/ql/target/hive-exec-2.2.0-SNAPSHOT.jar(org/apache/commons/lang3/ArrayUtils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/common/target/hive-common-2.2.0-SNAPSHOT.jar(org/apache/hadoop/hive/common/classification/InterfaceStability.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-hdfs/2.7.2/hadoop-hdfs-2.7.2.jar(org/apache/hadoop/hdfs/web/AuthFilter.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-2.2.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/Utils.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/security/UserGroupInformation.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar(org/apache/hadoop/security/authentication/client/PseudoAuthenticator.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-auth/2.7.2/hadoop-auth-2.7.2.jar(org/apache/hadoop/security/authentication/server/PseudoAuthenticationHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/2.7.2/hadoop-common-2.7.2.jar(org/apache/hadoop/util/GenericOptionsParser.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/rewrite/handler/RedirectPatternRule.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/rewrite/handler/RewriteHandler.class)]]
[loading
ZipFileIndexFileObject[/data/hiveptest/working/maven/org/eclipse/jetty/aggregate/jetty-all-server/7.6.0.v20120127/jetty-all-server-7.6.0.v20120127.jar(org/eclipse/jetty/server/Handler.class)]]
[loading
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905522#comment-15905522 ] Eugene Koifman commented on HIVE-15691: --- I don't think the 2.x version should have deprecated form of the c'tors. Since it's a new class no one could be using them yet. Otherwise I think this is OK for 2.x. Please update the patch and I can commit it. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904127#comment-15904127 ] Kalyan commented on HIVE-15691: --- Hi [~ekoifman], [~roshan_naik] current patch is only works on hive-2.x. if you feel looks ok, i will give the patch to work on hive-1.x currently flume need hive-1.x solution > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904081#comment-15904081 ] Eugene Koifman commented on HIVE-15691: --- If there are differences in API between 1.x and 2.x then I think we need 2 patches with 2.x deprecating the c'tors and 1.x not having the "new" c'tor - I think the patch as is won't even compile against Hive1. The build bot is only running gainst Hive2 > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904023#comment-15904023 ] Roshan Naik commented on HIVE-15691: [~ekoifman] looks like this has been idle for a bit. is there anything you need from [~kalyanhadoop] to move forward on this ? > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865168#comment-15865168 ] Roshan Naik commented on HIVE-15691: ok,. Flume is not yet switched over to 2.x Hive. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865077#comment-15865077 ] Eugene Koifman commented on HIVE-15691: --- Actually, it's the version of c'tor that takes a StreamingConnection parameter is the new/better version. Note that in other Writers the c'tors w/o connection parameter are all deprecated > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864938#comment-15864938 ] Kalyan commented on HIVE-15691: --- Thanks for your support, [~roshan_naik], [~ekoifman] To work with flume, regex support we are adding `StrictRegexWriter`. If this patch only for latest version 2.x, i need only 2 constructors StrictRegexWriter(String regex, HiveEndPoint endPoint) StrictRegexWriter(String regex, HiveEndPoint endPoint, HiveConf conf) may be any committing on old version need other 2 constructors. As per your comments .. i will update the patch.. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864832#comment-15864832 ] Roshan Naik commented on HIVE-15691: I see. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864804#comment-15864804 ] Eugene Koifman commented on HIVE-15691: --- Actually, StreamingConnection is required. W/o that, in a Kerberos environment Connection and Writer end up with a different UserGroupInformation which cause issues. See HIVE-14114 > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864761#comment-15864761 ] Roshan Naik commented on HIVE-15691: [~ekoifman] & [~kalyanhadoop] 1) I think the class needs only 2 constructors: - StrictRegexWriter(String regex, HiveEndPoint endPoint) - StrictRegexWriter(String regex, HiveEndPoint endPoint, HiveConf conf) The 'connection' param should be eliminated. 2) I cannot say much about the correctness of createSerde() & encode() methods, as i have not worked with RegexSerDe. Would be good to know if this has been validated via a manual test where the data streamed via this writer, was properly queryable from Hive. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15861858#comment-15861858 ] Roshan Naik commented on HIVE-15691: Should be able to get to it on Monday. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858643#comment-15858643 ] Eugene Koifman commented on HIVE-15691: --- [~roshan_naik], I was asking for your comments on the actual patch since you are the original author this API > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858601#comment-15858601 ] Roshan Naik commented on HIVE-15691: StrictRegexWriter allows extracting fields from the incoming text based on user specified regexes. DelimitedWriter is for extracting fields from things like comma/tab separated text. My understanding is that, implementation wise, this is closely modeled around the DelimitedWriter. [~kalyanhadoop] can better speak to the implementation details. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857108#comment-15857108 ] Eugene Koifman commented on HIVE-15691: --- What is main use case this is designed for that DelimitedWriter can't handle? Using different delimiters in the same row? Maybe the unit tests should be more elaborate to illustrate that. Why are there so many StrictRegexWriter() c'tors? It seems like a StrictRegexWriter w/o a regex or EndPoint or Connection, etc is not useful - it will just NPE in various places. > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[ https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852773#comment-15852773 ] Kalyan commented on HIVE-15691: --- Hi [~ekoifman], can please review the above patch Thanks > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink > - > > Key: HIVE-15691 > URL: https://issues.apache.org/jira/browse/HIVE-15691 > Project: Hive > Issue Type: New Feature > Components: HCatalog, Transactions >Reporter: Kalyan >Assignee: Kalyan > Attachments: HIVE-15691.1.patch, HIVE-15691.patch, > HIVE-15691-updated.patch > > > Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink. > It is similar to StrictJsonWriter available in hive. > Dependency is there in flume to commit. > FLUME-3036 : Create a RegexSerializer for Hive Sink. > Patch is available for Flume, Please verify the below link > https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15691) Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
[
https://issues.apache.org/jira/browse/HIVE-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833665#comment-15833665
]
Hive QA commented on HIVE-15691:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12848783/HIVE-15691.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10979 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=136)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[offset_limit_ppd_optimizer]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple]
(batchId=152)
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_Regex
(batchId=182)
org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchCommit_RegexUGI
(batchId=182)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3117/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3117/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3117/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12848783 - PreCommit-HIVE-Build
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink
> -
>
> Key: HIVE-15691
> URL: https://issues.apache.org/jira/browse/HIVE-15691
> Project: Hive
> Issue Type: New Feature
> Components: HCatalog, Transactions
>Reporter: Kalyan
>Assignee: Kalyan
> Attachments: HIVE-15691.patch
>
>
> Create StrictRegexWriter to work with RegexSerializer for Flume Hive Sink.
> It is similar to StrictJsonWriter available in hive.
> Dependency is there in flume to commit.
> FLUME-3036 : Create a RegexSerializer for Hive Sink.
> Patch is available for Flume, Please verify the below link
> https://github.com/kalyanhadooptraining/flume/commit/1c651e81395404321f9964c8d9d2af6f4a2aaef9
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
