[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528364#comment-15528364 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user asfgit closed the pull request at: https://github.com/apache/incubator-gearpump/pull/86 > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528178#comment-15528178 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/86 +1 > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526093#comment-15526093
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r80688415
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -19,27 +19,33 @@ package org.apache.gearpump.external.hbase
import java.io.{File, ObjectInputStream, ObjectOutputStream}
+import org.apache.gearpump.Message
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.streaming.sink.DataSink
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.gearpump.util.{Constants, FileUtils}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{Connection, ConnectionFactory, Put}
+import org.apache.hadoop.hbase.security.UserProvider
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
-import org.apache.hadoop.hbase.security.{User, UserProvider}
import org.apache.hadoop.security.UserGroupInformation
-import org.apache.gearpump.Message
-import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.sink.DataSink
-import org.apache.gearpump.streaming.task.TaskContext
-import org.apache.gearpump.util.{Constants, FileUtils}
+class HBaseSink(userconfig: UserConfig, tableName: String,
+val conn: (UserConfig, Configuration)
+=> Connection, @transient var configuration: Configuration)
+ extends DataSink {
-class HBaseSink(
-userconfig: UserConfig, tableName: String, @transient var
configuration: Configuration)
- extends DataSink{
- lazy val connection = HBaseSink.getConnection(userconfig, configuration)
+ lazy val connection = conn(userconfig, configuration)
lazy val table = connection.getTable(TableName.valueOf(tableName))
override def open(context: TaskContext): Unit = {}
+ def this(userconfig: UserConfig, tableName: String, configuration:
Configuration) = {
+this(userconfig, tableName, (userconfig: UserConfig, config:
Configuration) =>
+ {HBaseSink.getConnection(userconfig, config)}, configuration)
--- End diff --
just passing in `getConnection` will do, I think
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515219#comment-15515219 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/86 One question, now the `connection` is a transient value, will there be a NPE after the HBaseSink deserialized? > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515213#comment-15515213
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r80175062
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -115,15 +116,24 @@ object HBaseSink {
val COLUMN_NAME = "hbase.table.column.name"
val HBASE_USER = "hbase.user"
- def apply[T](userconfig: UserConfig, tableName: String): HBaseSink = {
-new HBaseSink(userconfig, tableName)
- }
def apply[T](userconfig: UserConfig, tableName: String, configuration:
Configuration)
-: HBaseSink = {
+ : HBaseSink = {
new HBaseSink(userconfig, tableName, configuration)
}
+ def apply[T](
--- End diff --
Code format
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515173#comment-15515173 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/86 @huafengw any more comments ? > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515019#comment-15515019
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r80167763
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -114,16 +118,19 @@ object HBaseSink {
val COLUMN_FAMILY = "hbase.table.column.family"
val COLUMN_NAME = "hbase.table.column.name"
val HBASE_USER = "hbase.user"
-
- def apply[T](userconfig: UserConfig, tableName: String): HBaseSink = {
-new HBaseSink(userconfig, tableName)
- }
+ private val CONFIG = HBaseConfiguration.create()
--- End diff --
will configuration be created at client side ?
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512959#comment-15512959 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user manuzhang commented on a diff in the pull request: https://github.com/apache/incubator-gearpump/pull/86#discussion_r80017412 --- Diff: external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala --- @@ -97,7 +101,7 @@ class HBaseSink( } /** - * Overrides Java's default deserialization + * Overrides Java's default serialization --- End diff -- serialization ? > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512957#comment-15512957
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r80017362
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -19,29 +19,33 @@ package org.apache.gearpump.external.hbase
import java.io.{File, ObjectInputStream, ObjectOutputStream}
+import org.apache.gearpump.Message
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.streaming.sink.DataSink
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.gearpump.util.{Constants, FileUtils}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{Connection, ConnectionFactory, Put}
+import org.apache.hadoop.hbase.security.UserProvider
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
-import org.apache.hadoop.hbase.security.{User, UserProvider}
import org.apache.hadoop.security.UserGroupInformation
-import org.apache.gearpump.Message
-import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.sink.DataSink
-import org.apache.gearpump.streaming.task.TaskContext
-import org.apache.gearpump.util.{Constants, FileUtils}
+class HBaseSink(userconfig: UserConfig, tableName: String, @transient var
connection: Connection,
+@transient var configuration: Configuration)
+ extends DataSink {
-class HBaseSink(
-userconfig: UserConfig, tableName: String, @transient var
configuration: Configuration)
- extends DataSink{
- lazy val connection = HBaseSink.getConnection(userconfig, configuration)
lazy val table = connection.getTable(TableName.valueOf(tableName))
override def open(context: TaskContext): Unit = {}
def this(userconfig: UserConfig, tableName: String) = {
-this(userconfig, tableName, HBaseConfiguration.create())
+this(userconfig, tableName, HBaseSink.getConnection(userconfig,
HBaseConfiguration.create()),
--- End diff --
why create configuration twice here ?
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512302#comment-15512302 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/86 @huafengw the current `HBaseSink` constructor is not testable. I'm afraid changing it is the only way to go. > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512297#comment-15512297
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r79979616
--- Diff: project/BuildExample.scala ---
@@ -57,6 +57,7 @@ object BuildExample extends sbt.Build {
)
) dependsOn(streaming % "test->test; provided", daemon % "test->test;
provided")
+
--- End diff --
unnecessary changes
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512294#comment-15512294
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r79979548
--- Diff:
external/hbase/src/test/scala/org/apache/gearpump/external/hbase/HBaseSinkSpec.scala
---
@@ -17,24 +17,52 @@
*/
package org.apache.gearpump.external.hbase
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.streaming.MockUtil
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.TableName
+import org.apache.hadoop.hbase.client._
+import org.apache.hadoop.hbase.util.Bytes
+import org.mockito.Mockito._
+import org.scalatest.mock.MockitoSugar
import org.scalatest.prop.PropertyChecks
import org.scalatest.{Matchers, PropSpec}
-class HBaseSinkSpec extends PropSpec with PropertyChecks with Matchers {
+class HBaseSinkSpec extends PropSpec with PropertyChecks with Matchers
with MockitoSugar {
property("HBaseSink should insert a row successfully") {
- // import Mockito._
- // val htable = Mockito.mock(classOf[HTable])
- // val row = "row"
- // val group = "group"
- // val name = "name"
- // val value = "1.2"
- // val put = new Put(Bytes.toBytes(row))
- // put.add(Bytes.toBytes(group), Bytes.toBytes(name),
Bytes.toBytes(value))
- // val hbaseSink = HBaseSink(htable)
- // hbaseSink.insert(put)
- // verify(htable).put(put)
+val table = mock[Table]
+val config = mock[Configuration]
+val conn = mock[Connection]
+val taskContext = mock[TaskContext]
+
+
+val map = Map[String, String]("HBASESINK" -> "hbasesink", "TABLE_NAME"
-> "hbase.table.name",
+ "COLUMN_FAMILY" -> "hbase.table.column.family", "COLUMN_NAME" ->
"hbase.table.column.name",
+ "HBASE_USER" -> "hbase.user", "GEARPUMP_KERBEROS_PRINCIPAL" ->
"gearpump.kerberos.principal",
+ "GEARPUMP_KEYTAB_FILE" -> "gearpump.keytab.file"
+)
+val userconfig = new UserConfig(map)
+val tablename = "hbase"
+val row = "row"
+val group = "group"
+val name = "name"
+val value = "1.2"
+
+when(conn.getTable(TableName.valueOf(tablename))).thenReturn(table)
+
+val connection = HBaseSink.getConn(conn, true, userconfig, config)
--- End diff --
simply use mocked `conn` here
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512291#comment-15512291
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r79979402
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -148,11 +158,24 @@ object HBaseSink {
val userName = userConfig.getString(HBASE_USER)
if (userName.isEmpty) {
ConnectionFactory.createConnection(configuration)
+
} else {
val user = UserProvider.instantiate(configuration)
.create(UserGroupInformation.createRemoteUser(userName.get))
ConnectionFactory.createConnection(configuration, user)
}
+
+ }
+ def getConn(connection: Connection, isTest: Boolean, userConfig:
UserConfig,
--- End diff --
We don't need `isTest` here since this method is not supposed to be called
in test
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512285#comment-15512285
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r79979330
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -19,39 +19,47 @@ package org.apache.gearpump.external.hbase
import java.io.{File, ObjectInputStream, ObjectOutputStream}
+import org.apache.gearpump.Message
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.streaming.sink.DataSink
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.gearpump.util.{Constants, FileUtils}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{Connection, ConnectionFactory, Put}
+import org.apache.hadoop.hbase.security.UserProvider
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
-import org.apache.hadoop.hbase.security.{User, UserProvider}
import org.apache.hadoop.security.UserGroupInformation
-import org.apache.gearpump.Message
-import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.sink.DataSink
-import org.apache.gearpump.streaming.task.TaskContext
-import org.apache.gearpump.util.{Constants, FileUtils}
+class HBaseSink(userconfig: UserConfig, tableName: String, @transient var
connection: Connection,
+@transient var configuration: Configuration)
+ extends DataSink {
-class HBaseSink(
-userconfig: UserConfig, tableName: String, @transient var
configuration: Configuration)
- extends DataSink{
- lazy val connection = HBaseSink.getConnection(userconfig, configuration)
+
+ // var connection = HBaseSink.getConnection(userconfig, configuration)
--- End diff --
to allow it be mocked in test
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512276#comment-15512276
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r79979033
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -19,39 +19,47 @@ package org.apache.gearpump.external.hbase
import java.io.{File, ObjectInputStream, ObjectOutputStream}
+import org.apache.gearpump.Message
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.streaming.sink.DataSink
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.gearpump.util.{Constants, FileUtils}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{Connection, ConnectionFactory, Put}
+import org.apache.hadoop.hbase.security.UserProvider
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
-import org.apache.hadoop.hbase.security.{User, UserProvider}
import org.apache.hadoop.security.UserGroupInformation
-import org.apache.gearpump.Message
-import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.sink.DataSink
-import org.apache.gearpump.streaming.task.TaskContext
-import org.apache.gearpump.util.{Constants, FileUtils}
+class HBaseSink(userconfig: UserConfig, tableName: String, @transient var
connection: Connection,
+@transient var configuration: Configuration)
+ extends DataSink {
-class HBaseSink(
-userconfig: UserConfig, tableName: String, @transient var
configuration: Configuration)
- extends DataSink{
- lazy val connection = HBaseSink.getConnection(userconfig, configuration)
+
+ // var connection = HBaseSink.getConnection(userconfig, configuration)
lazy val table = connection.getTable(TableName.valueOf(tableName))
override def open(context: TaskContext): Unit = {}
def this(userconfig: UserConfig, tableName: String) = {
-this(userconfig, tableName, HBaseConfiguration.create())
+this(userconfig, tableName, HBaseSink.getConnection(userconfig,
HBaseConfiguration.create()),
+ HBaseConfiguration.create())
+ }
+ def this(userconfig: UserConfig, tableName: String, configuration:
Configuration) = {
+this(userconfig, tableName, HBaseSink.getConnection(userconfig,
HBaseConfiguration.create()),
+ HBaseConfiguration.create())
--- End diff --
`configuration` is passed in
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512273#comment-15512273
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user Roshanson commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r79978998
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -19,39 +19,47 @@ package org.apache.gearpump.external.hbase
import java.io.{File, ObjectInputStream, ObjectOutputStream}
+import org.apache.gearpump.Message
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.streaming.sink.DataSink
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.gearpump.util.{Constants, FileUtils}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{Connection, ConnectionFactory, Put}
+import org.apache.hadoop.hbase.security.UserProvider
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
-import org.apache.hadoop.hbase.security.{User, UserProvider}
import org.apache.hadoop.security.UserGroupInformation
-import org.apache.gearpump.Message
-import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.sink.DataSink
-import org.apache.gearpump.streaming.task.TaskContext
-import org.apache.gearpump.util.{Constants, FileUtils}
+class HBaseSink(userconfig: UserConfig, tableName: String, @transient var
connection: Connection,
+@transient var configuration: Configuration)
+ extends DataSink {
-class HBaseSink(
-userconfig: UserConfig, tableName: String, @transient var
configuration: Configuration)
- extends DataSink{
- lazy val connection = HBaseSink.getConnection(userconfig, configuration)
+
+ // var connection = HBaseSink.getConnection(userconfig, configuration)
--- End diff --
I use the parameter of 'connection' to separate real connection to
HBase testing and unit testing.
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512262#comment-15512262 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user huafengw commented on the issue: https://github.com/apache/incubator-gearpump/pull/86 Generally my question is that is there any way implementing the HBaseSinkSpec without changing the HBaseSink? > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512251#comment-15512251
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user huafengw commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/86#discussion_r79978331
--- Diff:
external/hbase/src/main/scala/org/apache/gearpump/external/hbase/HBaseSink.scala
---
@@ -19,39 +19,47 @@ package org.apache.gearpump.external.hbase
import java.io.{File, ObjectInputStream, ObjectOutputStream}
+import org.apache.gearpump.Message
+import org.apache.gearpump.cluster.UserConfig
+import org.apache.gearpump.streaming.sink.DataSink
+import org.apache.gearpump.streaming.task.TaskContext
+import org.apache.gearpump.util.{Constants, FileUtils}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.hbase.client.{Connection, ConnectionFactory, Put}
+import org.apache.hadoop.hbase.security.UserProvider
import org.apache.hadoop.hbase.util.Bytes
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
-import org.apache.hadoop.hbase.security.{User, UserProvider}
import org.apache.hadoop.security.UserGroupInformation
-import org.apache.gearpump.Message
-import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.sink.DataSink
-import org.apache.gearpump.streaming.task.TaskContext
-import org.apache.gearpump.util.{Constants, FileUtils}
+class HBaseSink(userconfig: UserConfig, tableName: String, @transient var
connection: Connection,
+@transient var configuration: Configuration)
+ extends DataSink {
-class HBaseSink(
-userconfig: UserConfig, tableName: String, @transient var
configuration: Configuration)
- extends DataSink{
- lazy val connection = HBaseSink.getConnection(userconfig, configuration)
+
+ // var connection = HBaseSink.getConnection(userconfig, configuration)
--- End diff --
Why change the `connection`?
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512244#comment-15512244 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user codecov-io commented on the issue: https://github.com/apache/incubator-gearpump/pull/86 ## [Current coverage](https://codecov.io/gh/apache/incubator-gearpump/pull/86?src=pr) is 70.63% (diff: 22.22%) > Merging [#86](https://codecov.io/gh/apache/incubator-gearpump/pull/86?src=pr) into [master](https://codecov.io/gh/apache/incubator-gearpump/branch/master?src=pr) will decrease coverage by **0.17%** ```diff @@ master#86 diff @@ == Files 178178 Lines 5903 5909 +6 Methods5584 5403 -181 Messages 0 0 Branches319506 +187 == - Hits 4180 4174 -6 - Misses 1723 1735+12 Partials 0 0 ```  > Powered by [Codecov](https://codecov.io?src=pr). Last update [5cd5b93...3086e45](https://codecov.io/gh/apache/incubator-gearpump/compare/5cd5b9304f0f703c6f89d2865e5bb7adbb631c74...3086e4535fbbaa1c6e652f8e3cc3fdeb7bf24ead?src=pr) > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512184#comment-15512184 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user Roshanson closed the pull request at: https://github.com/apache/incubator-gearpump/pull/84 > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509649#comment-15509649 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/84 This PR contains commits for GEARPUMP-192. Please rebase. > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[
https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509647#comment-15509647
]
ASF GitHub Bot commented on GEARPUMP-204:
-
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/84#discussion_r79813470
--- Diff: conf/gear.conf ---
@@ -243,7 +243,7 @@ gearpump {
###
executor {
vmargs = "-server -Xms512M -Xmx1024M -Xss1M
-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3
-Djava.rmi.server.hostname=localhost"
-extraClasspath = ""
+extraClasspath =
"/home/manuzhang/hbase-1.2.2/lib/*:/home/manuzhang/hbase-1.2.2/conf"
--- End diff --
please remove this specific configuration
> Add unit test for external_hbase module
> ---
>
> Key: GEARPUMP-204
> URL: https://issues.apache.org/jira/browse/GEARPUMP-204
> Project: Apache Gearpump
> Issue Type: Improvement
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508795#comment-15508795 ] ASF GitHub Bot commented on GEARPUMP-204: - Github user manuzhang commented on the issue: https://github.com/apache/incubator-gearpump/pull/84 please rebase on latest master and remove test examples and configurations > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (GEARPUMP-204) Add unit test for external_hbase module
[ https://issues.apache.org/jira/browse/GEARPUMP-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508722#comment-15508722 ] ASF GitHub Bot commented on GEARPUMP-204: - GitHub user Roshanson opened a pull request: https://github.com/apache/incubator-gearpump/pull/84 Fix [GEARPUMP-204] Add unit test for external_hbase module Fix [GEARPUMP-204] Add unit test for external_hbase module You can merge this pull request into a Git repository by running: $ git pull https://github.com/Roshanson/incubator-gearpump fix-tohbase Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-gearpump/pull/84.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #84 commit e70cb18ec8dff8e59448853e32cb9acbf1696c81 Author: Roshanson <736781...@qq.com> Date: 2016-08-26T07:08:07Z refactor example sources task to use DataSourceAPI commit ff3d1401c307d661307138ff05c29c9f03bc3961 Author: Roshanson <736781...@qq.com> Date: 2016-08-29T11:32:16Z refactor example sources task to use DataSourceAPI commit 4ff8453c494773b39acadceb9848aecbdd432859 Author: Roshanson <736781...@qq.com> Date: 2016-09-02T11:17:33Z refactor example sources task to use DataSourceAPI commit d8f0838ed3e31b844db46e122df56658a0a698a9 Author: 736781...@qq.com Date: 2016-09-02T12:24:30Z refactor example sources task to use DataSourceAPI commit bc9a2ee818278abc7b0b4bd8b3a42026f3905236 Author: 736781...@qq.com Date: 2016-09-02T12:26:48Z refactor example sources task to use DataSourceAPI commit 7191000bd4a97edb6575b092265ca886a61e07b0 Author: 736781...@qq.com Date: 2016-09-02T12:31:28Z refactor example sources task to use DataSourceAPI commit f4c6737198cc0a0a9a1a4ae42b6aece1e6a4eb3c Author: 736781...@qq.com Date: 2016-09-02T13:16:16Z [GEARPUMP-192] Refactor example sources task to use DataSourceAPI commit 84e884c42196a091d170a175600a91a4858d8ad4 Author: Roshanson <736781...@qq.com> Date: 2016-09-02T13:25:43Z [GEARPUMP-192] Refactor example sources task to use DataSourceAPI commit ee7d2acdb50686e781b31c283bf651eeab64b155 Author: Roshanson <736781...@qq.com> Date: 2016-09-13T13:30:57Z [GEARPUMP-204]Add unit test for external_hbase module commit 65e13bde0d728da62396ac59c73d01b3c5dc1320 Author: roshanson Date: 2016-09-19T09:00:49Z connection to hbase commit 872991e65e1904358e81d5a15757e7accfaa734c Author: roshanson Date: 2016-09-21T04:33:57Z [GEARPUMP-204]add unit test for external_hbase module > Add unit test for external_hbase module > --- > > Key: GEARPUMP-204 > URL: https://issues.apache.org/jira/browse/GEARPUMP-204 > Project: Apache Gearpump > Issue Type: Improvement >Reporter: Manu Zhang >Assignee: Kaifang Bao > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
