[GitHub] sijie commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

sijie commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152196993
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] sijie commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

sijie commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152196627
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] sijie commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

sijie commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152196082
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] sijie commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

sijie commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152195920
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] sijie commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

sijie commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152195477
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

Jenkins build became unstable: distributedlog-nightly-build #493

[Apache Jenkins Server]

See 

[GitHub] sijie commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

sijie commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152194844
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -277,122 +281,13 @@ private void checkEnvironment(RegistrationManager rm) 
throws BookieException, IO
 return;
 }
 
-if (conf.getAllowStorageExpansion()) {
-checkEnvironmentWithStorageExpansion(conf, rm, journalDirectories, 
allLedgerDirs);
-return;
-}
-
-try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
-}
-
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
-try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
-// If allowStorageExpansion option is set, we should
-// make sure that the new set of ledger/index dirs
-// is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verify(journalCookie);
-}
-}
-
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
-masterCookie.verify(c);
-} catch (FileNotFoundException fnf) {
-missedCookieDirs.add(dir);
-}
-}
-
-if (!newEnv && missedCookieDirs.size() > 0) {
-// If we find that any of the dirs in missedCookieDirs, existed
-// previously, we stop because we could be missing data
-// Also, if a new ledger dir is being added, we make sure that
-// that dir is empty. Else, we reject the request
-Set existingLedgerDirs = Sets.newHashSet();
-for (Cookie journalCookie : journalCookies) {
-Collections.addAll(existingLedgerDirs, 
journalCookie.getLedgerDirPathsFromCookie());
-}
-List dirsMissingData = new ArrayList();
-List nonEmptyDirs = new ArrayList();
-for (File dir : missedCookieDirs) {
-if (existingLedgerDirs.contains(dir.getParent())) {
-// if one of the existing ledger dirs doesn't have 
cookie,
-// let us not proceed further
-dirsMissingData.add(dir);
-continue;
-}
-String[] content = dir.list();
-if (content != null && content.length != 0) {
-nonEmptyDirs.add(dir);
-}
-}
-if (dirsMissingData.size() > 0 || nonEmptyDirs.size() > 0) {
-LOG.error("Either not all local directories have cookies 
or directories being added "
-+ " newly are not empty. "
-+ "Directories missing cookie file are: " + 
dirsMissingData
-+ " New directories that are not empty are: " + 
nonEmptyDirs);
-throw new BookieException.InvalidCookieException();
-}
-}
-
-  

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193201
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -277,122 +281,13 @@ private void checkEnvironment(RegistrationManager rm) 
throws BookieException, IO
 return;
 }
 
-if (conf.getAllowStorageExpansion()) {
-checkEnvironmentWithStorageExpansion(conf, rm, journalDirectories, 
allLedgerDirs);
-return;
-}
-
-try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
-}
-
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
-try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
-// If allowStorageExpansion option is set, we should
-// make sure that the new set of ledger/index dirs
-// is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verify(journalCookie);
-}
-}
-
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
-masterCookie.verify(c);
-} catch (FileNotFoundException fnf) {
-missedCookieDirs.add(dir);
-}
-}
-
-if (!newEnv && missedCookieDirs.size() > 0) {
-// If we find that any of the dirs in missedCookieDirs, existed
-// previously, we stop because we could be missing data
-// Also, if a new ledger dir is being added, we make sure that
-// that dir is empty. Else, we reject the request
-Set existingLedgerDirs = Sets.newHashSet();
-for (Cookie journalCookie : journalCookies) {
-Collections.addAll(existingLedgerDirs, 
journalCookie.getLedgerDirPathsFromCookie());
-}
-List dirsMissingData = new ArrayList();
-List nonEmptyDirs = new ArrayList();
-for (File dir : missedCookieDirs) {
-if (existingLedgerDirs.contains(dir.getParent())) {
-// if one of the existing ledger dirs doesn't have 
cookie,
-// let us not proceed further
-dirsMissingData.add(dir);
-continue;
-}
-String[] content = dir.list();
-if (content != null && content.length != 0) {
-nonEmptyDirs.add(dir);
-}
-}
-if (dirsMissingData.size() > 0 || nonEmptyDirs.size() > 0) {
-LOG.error("Either not all local directories have cookies 
or directories being added "
-+ " newly are not empty. "
-+ "Directories missing cookie file are: " + 
dirsMissingData
-+ " New directories that are not empty are: " + 
nonEmptyDirs);
-throw new BookieException.InvalidCookieException();
-}
-}
-
-  

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193185
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193172
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193166
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193139
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193123
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193148
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193129
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193092
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jvrao commented on a change in pull request #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

jvrao commented on a change in pull request #712: Issue 544: Bootup cookie 
validation considers an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#discussion_r152193112
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
 ##
 @@ -441,117 +336,220 @@ private void checkIfDirsOnSameDiskPartition(List 
dirs) throws DiskPartitio
 }
 }
 
-public static void checkEnvironmentWithStorageExpansion(
-ServerConfiguration conf,
-RegistrationManager rm,
-List journalDirectories,
-List allLedgerDirs) throws BookieException {
+static List possibleBookieIds(ServerConfiguration 
conf)
+throws BookieException {
+// we need to loop through all possible bookie identifiers to ensure 
it is treated as a new environment
+// just because of bad configuration
+List addresses = 
Lists.newArrayListWithExpectedSize(3);
 try {
-boolean newEnv = false;
-List missedCookieDirs = new ArrayList();
-List journalCookies = Lists.newArrayList();
-// try to read cookie from journal directory.
-for (File journalDirectory : journalDirectories) {
-try {
-Cookie journalCookie = 
Cookie.readFromDirectory(journalDirectory);
-journalCookies.add(journalCookie);
-if (journalCookie.isBookieHostCreatedFromIp()) {
-conf.setUseHostNameAsBookieID(false);
-} else {
-conf.setUseHostNameAsBookieID(true);
-}
-} catch (FileNotFoundException fnf) {
-newEnv = true;
-missedCookieDirs.add(journalDirectory);
-}
+// ip address
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(false).setAdvertisedAddress(null)));
+// host name
+addresses.add(getBookieAddress(
+new 
ServerConfiguration(conf).setUseHostNameAsBookieID(true).setAdvertisedAddress(null)));
+// advertised address
+if (null != conf.getAdvertisedAddress()) {
+addresses.add(getBookieAddress(conf));
 }
+} catch (UnknownHostException e) {
+throw new UnknownBookieIdException(e);
+}
+return addresses;
+}
 
-String instanceId = rm.getClusterInstanceId();
-Cookie.Builder builder = Cookie.generateCookie(conf);
-if (null != instanceId) {
-builder.setInstanceId(instanceId);
-}
-Cookie masterCookie = builder.build();
-Versioned rmCookie = null;
+static Versioned readAndVerifyCookieFromRegistrationManager(
+Cookie masterCookie, RegistrationManager rm,
+List addresses, boolean allowExpansion)
+throws BookieException {
+Versioned rmCookie = null;
+for (BookieSocketAddress address : addresses) {
 try {
-rmCookie = Cookie.readFromRegistrationManager(rm, conf);
+rmCookie = Cookie.readFromRegistrationManager(rm, address);
 // If allowStorageExpansion option is set, we should
 // make sure that the new set of ledger/index dirs
 // is a super set of the old; else, we fail the cookie check
-masterCookie.verifyIsSuperSet(rmCookie.getValue());
-} catch (CookieNotFoundException e) {
-// can occur in cases:
-// 1) new environment or
-// 2) done only metadata format and started bookie server.
-}
-for (File journalDirectory : journalDirectories) {
-checkDirectoryStructure(journalDirectory);
-}
-if (!newEnv) {
-for (Cookie journalCookie: journalCookies) {
-masterCookie.verifyIsSuperSet(journalCookie);
+if (allowExpansion) {
+masterCookie.verifyIsSuperSet(rmCookie.getValue());
+} else {
+masterCookie.verify(rmCookie.getValue());
 }
+} catch (CookieNotFoundException e) {
+continue;
 }
+}
+return rmCookie;
+}
 
-
-for (File dir : allLedgerDirs) {
-checkDirectoryStructure(dir);
-try {
-Cookie c = Cookie.readFromDirectory(dir);
+private static Pair verifyAndGetMissingDirs(
+Cookie masterCookie, boolean allowExpansion, List dirs)
+throws InvalidCookieException, IOException {
+List missingDirs = Lists.newArrayList();
+List existedCookies = Lists.newArrayList();
+

[GitHub] jiazhai commented on a change in pull request #738: Issue 731: refine LedgerEntry interface and implementation

[GitBox]

jiazhai commented on a change in pull request #738: Issue 731: refine 
LedgerEntry interface and implementation
URL: https://github.com/apache/bookkeeper/pull/738#discussion_r152178441
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/api/LedgerEntry.java
 ##
 @@ -56,11 +56,9 @@
 long getLength();
 
 /**
- * Returns the content of the entry. This method can be called only once. 
While using v2 wire protocol this method
- * will automatically release the internal ByteBuf.
+ * Returns the content of the entry.
  *
  * @return the content of the entry
- * @throws IllegalStateException if this method is called twice
  */
 byte[] getEntry();
 
 
 Review comment:
   In LedgerEntryImpl, LedgerEntry is reenterable,  closing is more for the 
ByteBuf release.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on a change in pull request #738: Issue 731: refine LedgerEntry interface and implementation

[GitBox]

sijie commented on a change in pull request #738: Issue 731: refine LedgerEntry 
interface and implementation
URL: https://github.com/apache/bookkeeper/pull/738#discussion_r152177922
 
 

 ##
 File path: 
bookkeeper-server/src/main/java/org/apache/bookkeeper/client/api/LedgerEntry.java
 ##
 @@ -32,7 +32,7 @@
  */
 @Public
 @Unstable
-public interface LedgerEntry {
+public interface LedgerEntry extends AutoCloseable {
 
 Review comment:
   @ivankelly : a good point. it was inconsistent there because I asked 
@eolivelli to not use `byte[]`. but @eolivelli wants to have a `byte[]` array. 
that was probably what caused confusion in the initial commit. we can support 
all - `byte[]`, `ByteBuffer` and `ByteBuf`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] zhaijack opened a new issue #750: inconsistency in the data types used in read and write

[GitBox]

zhaijack opened a new issue #750:  inconsistency in the data types used in read 
and write
URL: https://github.com/apache/bookkeeper/issues/750
 
 
   
   **FEATURE REQUEST**
   
   1. Please describe the feature you are requesting.
   there is an inconsistency in the data types used in read and write.
   
   
   2. Indicate the importance of this issue to you (blocker, must-have, 
should-have, nice-to-have). Are you currently using any workarounds to address 
this issue?
   should-have.
   
   3. Provide any additional detail on your proposed use case for this feature.
   In PR #738 , Ivan pointed out: 
   ```
there is an inconsistency in the data types used in read and write.
   
   WriteHandle is takes ByteBuf and ByteBuffer
   ReadHandle/LedgerEntry returns ByteBuf and byte[]
   
   This should be resolved.
   ```
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jiazhai commented on issue #738: Issue 731: refine LedgerEntry interface and implementation

[GitBox]

jiazhai commented on issue #738: Issue 731: refine LedgerEntry interface and 
implementation
URL: https://github.com/apache/bookkeeper/pull/738#issuecomment-345913937
 
 
   updated this PR to keep old behavior of LedgerEntry; the tests that use 
client.LedgerEntry was un -touched.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #237: issue #236: shaded jar lose necessary bk jar to execute

[GitBox]

sijie closed pull request #237: issue #236: shaded jar lose necessary bk jar to 
execute
URL: https://github.com/apache/distributedlog/pull/237
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/distributedlog-core/pom.xml b/distributedlog-core/pom.xml
index f010959a..aa55c2e8 100644
--- a/distributedlog-core/pom.xml
+++ b/distributedlog-core/pom.xml
@@ -213,6 +213,7 @@
   commons-io:commons-io
   commons-lang:commons-lang
   com.google.guava:guava
+  com.google.protobuf:protobuf-java
   io.netty:netty
   io.netty:netty-all
   io.netty:netty-buffer
@@ -223,6 +224,8 @@
   
org.apache.bookkeeper.stats:bookkeeper-stats-api
   org.apache.bookkeeper:bookkeeper-server
   org.apache.bookkeeper:bookkeeper-common
+  org.apache.bookkeeper:bookkeeper-proto
+  org.apache.bookkeeper:bookkeeper-http
   org.apache.commons:commons-lang3
   
org.apache.distributedlog:distributedlog-common
   
org.apache.distributedlog:distributedlog-protocol
@@ -340,6 +343,7 @@
   commons-io:commons-io
   commons-lang:commons-lang
   com.google.guava:guava
+  com.google.protobuf:protobuf-java
   io.netty:netty
   io.netty:netty-all
   io.netty:netty-buffer
@@ -349,6 +353,9 @@
   net.jpountz.lz4:lz4
   
org.apache.bookkeeper.stats:bookkeeper-stats-api
   org.apache.bookkeeper:bookkeeper-server
+  org.apache.bookkeeper:bookkeeper-common
+  org.apache.bookkeeper:bookkeeper-proto
+  org.apache.bookkeeper:bookkeeper-http
   org.apache.commons:commons-lang3
   
org.apache.distributedlog:distributedlog-common
   
org.apache.distributedlog:distributedlog-protocol


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed issue #236: shaded jar lose necessary bk jar to execute

[GitBox]

sijie closed issue #236: shaded jar lose necessary bk jar to execute
URL: https://github.com/apache/distributedlog/issues/236
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #749: Update release date for 4.5.1

[GitBox]

sijie commented on issue #749: Update release date for 4.5.1
URL: https://github.com/apache/bookkeeper/issues/749#issuecomment-345912462
 
 
   @eolivelli you need to update the release date for 4.5.1 after it is 
announced.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new issue #749: Update release date for 4.5.1

[GitBox]

sijie opened a new issue #749: Update release date for 4.5.1
URL: https://github.com/apache/bookkeeper/issues/749
 
 
   we need to update the release date for `4.5.1` in `releases.md` when 4.5.1 
is announced.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed pull request #748: Issue 747: Reorder releases menu - 4.5.1 should appear before 4.5.0

[GitBox]

sijie closed pull request #748: Issue 747: Reorder releases menu - 4.5.1 should 
appear before 4.5.0
URL: https://github.com/apache/bookkeeper/pull/748
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/site/_config.yml b/site/_config.yml
index 7ff3e54be..7c305e4b0 100644
--- a/site/_config.yml
+++ b/site/_config.yml
@@ -10,9 +10,9 @@ twitter_url: https://twitter.com/asfbookkeeper
 livereload: true
 
 versions:
-- "4.5.0"
-- "4.5.1"
 # [next_version_placehodler]
+- "4.5.1"
+- "4.5.0"
 
 archived_versions:
 - "4.4.0"
@@ -29,7 +29,7 @@ archived_versions:
 latest_version: "4.6.0-SNAPSHOT"
 latest_release: "4.5.1"
 stable_release: "4.5.0"
-distributedlog_version: "2.1.0-0.4.0"
+distributedlog_version: "0.5.0"
 
 defaults:
 - scope:


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie closed issue #747: Reorder releases menu, 4.5.0 must be after 4.5.1

[GitBox]

sijie closed issue #747: Reorder releases menu, 4.5.0 must be after 4.5.1
URL: https://github.com/apache/bookkeeper/issues/747
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie opened a new pull request #748: Issue 747: Reorder releases menu - 4.5.1 should appear before 4.5.0

[GitBox]

sijie opened a new pull request #748: Issue 747: Reorder releases menu - 4.5.1 
should appear before 4.5.0
URL: https://github.com/apache/bookkeeper/pull/748
 
 
   Descriptions of the changes in this PR:
   
   - 4.5.1 should appear before 4.5.0
   - bump dlog version to 0.5.0
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] eolivelli opened a new issue #747: Reorder releases menu, 4.5.1 must be after 4.5.0

[GitBox]

eolivelli opened a new issue #747: Reorder releases menu, 4.5.1 must be after 
4.5.0
URL: https://github.com/apache/bookkeeper/issues/747
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #738: Issue 731: refine LedgerEntry interface and implementation

[GitBox]

sijie commented on issue #738: Issue 731: refine LedgerEntry interface and 
implementation
URL: https://github.com/apache/bookkeeper/pull/738#issuecomment-345783089
 
 
   @ivankelly that is the 4.5 behavior of LedgerEntry. in the new api, we 
should change LedgerEntry to be a more consistent behavior.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Jenkins build is back to normal : bookkeeper-master #1949

[Apache Jenkins Server]

See 

[GitHub] eolivelli closed pull request #746: Issue-732 Update Dockerfile for 4.5.1 version

[GitBox]

eolivelli closed pull request #746: Issue-732 Update Dockerfile for 4.5.1 
version
URL: https://github.com/apache/bookkeeper/pull/746
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/docker/Dockerfile b/docker/Dockerfile
index 2af5a61ea..59538bb30 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -20,9 +20,9 @@
 FROM centos:7
 MAINTAINER Apache BookKeeper 
 
-ARG BK_VERSION=4.5.0
+ARG BK_VERSION=4.5.1
 ARG DISTRO_NAME=bookkeeper-server-${BK_VERSION}-bin
-ARG GPG_KEY=FD74402C
+ARG GPG_KEY=A615D22C
 
 ENV BOOKIE_PORT=3181
 EXPOSE $BOOKIE_PORT
diff --git a/docker/Makefile b/docker/Makefile
index 23a3ecf46..2b75519bd 100644
--- a/docker/Makefile
+++ b/docker/Makefile
@@ -83,6 +83,7 @@ run-bk:
--name "$(CONTAINER_NAME)" \
--hostname "$(CONTAINER_NAME)" \
--env BK_zkServers=$(ZK_CONTAINER_NAME):2181 \
+   --env 
BK_statsProviderClass=org.apache.bookkeeper.stats.NullStatsProvider \
--env BK_zkLedgersRootPath=$(BK_zkLedgersRootPath) \
$(IMAGE)
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jiazhai commented on a change in pull request #737: Issue 688: add bookkeeper-all package and update notice files

[GitBox]

jiazhai commented on a change in pull request #737:  Issue 688: add 
bookkeeper-all package and update notice files
URL: https://github.com/apache/bookkeeper/pull/737#discussion_r151976042
 
 

 ##
 File path: bookkeeper-dist/src/assemble/bin-all.xml
 ##
 @@ -0,0 +1,100 @@
+
+http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2
 http://maven.apache.org/xsd/assembly-1.1.2.xsd;>
+  bin
+  
+tar.gz
+  
+  true
+  
+
+  target
+  /
+  
+${project.artifactId}-${project.version}.jar
+  
+
+
+  ../../bookkeeper-server/conf
+  /conf
+
+
+  ../../bookkeeper-server/bin
 
 Review comment:
   It is strange that, in my verify, the bin/bookkeeper for dist is not the 
same as bookkeeper-server/bin/bookkeeper
   ```
   ?  bookkeeper git:(sijie/add_slim_module) ll 
bookkeeper-dist/target/bookkeeper-dist-4.6.0-SNAPSHOT/bin 
   total 40
   -rwxr-xr-x  1 jia  staff   7.0K Oct 17 15:33 bookkeeper
   -rwxr-xr-x  1 jia  staff   2.8K May 27 21:21 bookkeeper-cluster.sh
   -rwxr-xr-x  1 jia  staff   4.3K May 27 21:21 bookkeeper-daemon.sh
   ?  bookkeeper git:(sijie/add_slim_module) ll bookkeeper-server/bin 
   total 40
   -rwxr-xr-x  1 jia  staff   7.2K Nov 20 19:30 bookkeeper
   -rw-r--r--  1 jia  staff   2.8K May 27 21:21 bookkeeper-cluster.sh
   -rwxr-xr-x  1 jia  staff   4.3K May 27 21:21 bookkeeper-daemon.sh
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] sijie commented on issue #677: Issue 659: Fix Checkpoint logic in SortedLedgerStorage

[GitBox]

sijie commented on issue #677: Issue 659: Fix Checkpoint logic in 
SortedLedgerStorage
URL: https://github.com/apache/bookkeeper/pull/677#issuecomment-345653926
 
 
   @jvrao @ivankelly @reddycharan please take a look when you have time.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jiazhai commented on issue #737: Issue 688: add bookkeeper-all package and update notice files

[GitBox]

jiazhai commented on issue #737:  Issue 688: add bookkeeper-all package and 
update notice files
URL: https://github.com/apache/bookkeeper/pull/737#issuecomment-345631830
 
 
   +1 for the Integration test.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Jenkins build is still unstable: distributedlog-release-nightly-snapshot #85

[Apache Jenkins Server]

See 

[GitHub] asfgit commented on issue #712: Issue 544: Bootup cookie validation considers an empty journal to signify a new bookie

[GitBox]

asfgit commented on issue #712: Issue 544: Bootup cookie validation considers 
an empty journal to signify a new bookie
URL: https://github.com/apache/bookkeeper/pull/712#issuecomment-345621880
 
 
   SUCCESS 

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Jenkins build is still unstable: bookkeeper-release-nightly-snapshot #108

[Apache Jenkins Server]

See 

[GitHub] eolivelli commented on a change in pull request #737: Issue 688: add bookkeeper-all package and update notice files

[GitBox]

eolivelli commented on a change in pull request #737:  Issue 688: add 
bookkeeper-all package and update notice files
URL: https://github.com/apache/bookkeeper/pull/737#discussion_r151921043
 
 

 ##
 File path: bookkeeper-server/bin/bookkeeper
 ##
 @@ -71,13 +71,18 @@ else
 fi
 
 # exclude tests jar
-RELEASE_JAR=$(ls ${BK_HOME}/bookkeeper-server-*.jar 2> /dev/null | grep -v 
tests | tail -1)
+RELEASE_JAR=$(ls ${BK_HOME}/*bookkeeper-server-*.jar 2> /dev/null | grep -v 
tests | tail -1)
 
 Review comment:
   yes sure I had a bad idea in mind.
   usually you have the new package and you put the config files in order to 
use the data and the correct flags.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services